From 881957a463e1fd32458688e8bcff6b74f95d2ec7 Mon Sep 17 00:00:00 2001 From: Sandro Sp Date: Mon, 27 Apr 2026 23:10:51 -0700 Subject: [PATCH 001/286] [SPARK-55952][SPARK-55953][SQL] Add ResolveChangelogTable analyzer rule for batch CDC post-processing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is **PR 2 of a split** of #55426 (see the [split suggestion](https://github.com/apache/spark/pull/55426#issuecomment-4292375876) for the full plan). Its independent of PR 1, so we can merge in any order. For more context see [discussion](https://lists.apache.org/thread/dhxx6pohs7fvqc3knzhtoj4tbcgrwxts) posted to [devspark.apache.org](https://lists.apache.org/list.html?devspark.apache.org) and linked [SPIP](https://docs.google.com/document/d/1-4rCS3vsGIyhwnkAwPsEaqyUDg-AuVkdrYLotFPw0U0/edit?tab=t.0#heading=h.m1700lw4wsoj). Introduce the analyzer rule that post-processes a resolved `DataSourceV2Relation(ChangelogTable)` to inject carry-over removal and/or update detection, fused into a single pass over a `(rowId, _commit_version)`-partitioned Window. To prevent silent wrong results, it also includes an explicit rejection path for streaming CDC reads that would require post-processing. Included Changes: - `ResolveChangelogTable` analyzer rule: - **Batch**: applies the requested post-processing transformations. Carry-over removal is a `Filter` on the Window (drops CoW pairs where `min(rowVersion) == max(rowVersion)`). Update detection is a `CASE WHEN` over delete/insert counts (relabels pairs as `update_preimage` / `update_postimage`). The two passes are fused into a single Window. - **Streaming**: throws `INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED` when the requested options would need post-processing. Streams that don't need post-processing pass through unchanged. Actual streaming support is scoped to a follow-up PR. - **Net changes**: throws `INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED` for both batch and streaming. Actual implementation is scoped to a follow-up PR. - Option validation: throws `INVALID_CDC_OPTION.UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL` when `computeUpdates = true` is combined with a carry-over-surfacing connector and `deduplicationMode = none`, which would silently misclassify carry-overs as updates. - Runtime guard: the generated plan raises `INVALID_CDC_OPTION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION` when the connector emits more than one delete or insert for the same `(rowId, _commit_version)` partition, violating the `Changelog` contract. - `Analyzer`: register the rule after `ResolveRelations`. - `InMemoryChangelogCatalog`: `ChangelogProperties` extension so tests can configure post-processing scenarios without a real connector. ### Why are the changes needed? Currently, `CHANGES FROM VERSION ... WITH (deduplicationMode = ..., computeUpdates = ...)` parses the options, but they are silently ignored — connector output is returned raw. This PR wires the options to their actual semantics for batch reads, and prevents silent wrong results for streaming reads. ### Does this PR introduce _any_ user-facing change? Yes, for CDC queries against a `Changelog` connector.
Before/after example (click to expand) Given a `Changelog` connector that advertises both `containsCarryoverRows = true` and `representsUpdateAsDeleteAndInsert = true`, with rowId `id` and a `rowVersion` column, for versions 1–2: **Raw rows emitted by the connector:** ``` 1 | Alice | insert | 1 2 | Bob | insert | 1 3 | Carol | insert | 1 1 | Alice | delete | 2 -- part of rename Alice -> Alicia 1 | Alicia | insert | 2 -- part of rename Alice -> Alicia 2 | Bob | delete | 2 -- carry-over (CoW, row unchanged) 2 | Bob | insert | 2 -- carry-over (CoW, row unchanged) 3 | Carol | delete | 2 -- real delete ``` **Before this PR:** `WITH (computeUpdates = 'true')` is silently ignored, carry-overs leak through: ``` 1 | Alice | insert | 1 2 | Bob | insert | 1 3 | Carol | insert | 1 1 | Alice | delete | 2 1 | Alicia | insert | 2 2 | Bob | delete | 2 2 | Bob | insert | 2 3 | Carol | delete | 2 ``` **After this PR:** `WITH (computeUpdates = 'true')`: ``` 1 | Alice | insert | 1 2 | Bob | insert | 1 3 | Carol | insert | 1 1 | Alice | update_preimage | 2 1 | Alicia | update_postimage | 2 3 | Carol | delete | 2 ```
### How was this patch tested? `ResolveChangelogTablePostProcessingSuite` exercises the batch rule end-to-end via SQL against `InMemoryChangelogCatalog` (carry-over removal, update detection, their interaction across the option and connector-flag matrix, data-column handling with mixed types, and plan-shape invariants). `ChangelogResolutionSuite` adds streaming-rejection cases for the two capability flags that would require post-processing. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 Closes #55508 from SanJSp/SPARK-55668-PR2-resolve-changelog-table. Lead-authored-by: Sandro Sp Co-authored-by: Gengliang Wang Signed-off-by: Gengliang Wang --- .../resources/error/error-conditions.json | 28 + .../sql/connector/catalog/Changelog.java | 9 + .../sql/catalyst/analysis/Analyzer.scala | 1 + .../analysis/ResolveChangelogTable.scala | 312 +++++ .../sql/errors/QueryCompilationErrors.scala | 19 + .../datasources/v2/ChangelogTable.scala | 3 +- .../catalog/InMemoryChangelogCatalog.scala | 71 +- .../connector/ChangelogEndToEndSuite.scala | 26 +- .../connector/ChangelogResolutionSuite.scala | 76 +- ...lveChangelogTablePostProcessingSuite.scala | 1034 +++++++++++++++++ 10 files changed, 1549 insertions(+), 30 deletions(-) create mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json index e6c786640e090..ff34214e2ad95 100644 --- a/common/utils/src/main/resources/error/error-conditions.json +++ b/common/utils/src/main/resources/error/error-conditions.json @@ -661,6 +661,19 @@ ], "sqlState" : "42P08" }, + "CHANGELOG_CONTRACT_VIOLATION" : { + "message" : [ + "The Change Data Capture (CDC) connector violated the `Changelog` contract at runtime." + ], + "subClass" : { + "UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION" : { + "message" : [ + "Connector emitted multiple delete or insert rows for the same `(rowId, _commit_version)` partition. The `Changelog` contract requires at most one logical change per row identity per commit when `containsIntermediateChanges() = false`. Either fix the connector to deduplicate intermediate states, or set `containsIntermediateChanges() = true` and use `deduplicationMode = netChanges`." + ] + } + }, + "sqlState" : "XX000" + }, "CHECKPOINT_FILE_CHECKSUM_VERIFICATION_FAILED" : { "message" : [ "Checksum verification failed, the file may be corrupted. File: ", @@ -3278,6 +3291,21 @@ "message" : [ "`startingVersion` is required when `endingVersion` is specified for CDC queries." ] + }, + "NET_CHANGES_NOT_YET_SUPPORTED" : { + "message" : [ + "The `deduplicationMode = netChanges` option on connector `` is not yet supported. Use `deduplicationMode = dropCarryovers` (default) or `deduplicationMode = none` instead." + ] + }, + "STREAMING_POST_PROCESSING_NOT_SUPPORTED" : { + "message" : [ + "Change Data Capture (CDC) streaming reads on connector `` do not yet support post-processing (carry-over removal, update detection, or net change computation). The requested combination of options would require post-processing, which is currently only available for batch reads. Use a batch read, or set `deduplicationMode = none` and `computeUpdates = false` to receive raw change rows in streaming." + ] + }, + "UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL" : { + "message" : [ + "`computeUpdates` cannot be used with `deduplicationMode=none` on connector `` because the connector emits copy-on-write carry-over pairs (`containsCarryoverRows()` returns true) that would be silently mislabeled as updates. Set `deduplicationMode` to `dropCarryovers` or `netChanges`." + ] } }, "sqlState" : "42K03" diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java index 0a811aa0ae4d7..5f2203aa1c379 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java @@ -43,6 +43,15 @@ @Evolving public interface Changelog { + /** Constant for the {@code _change_type} value of a row inserted into the table. */ + String CHANGE_TYPE_INSERT = "insert"; + /** Constant for the {@code _change_type} value of a row deleted from the table. */ + String CHANGE_TYPE_DELETE = "delete"; + /** Constant for the {@code _change_type} value of an update's pre-image row. */ + String CHANGE_TYPE_UPDATE_PREIMAGE = "update_preimage"; + /** Constant for the {@code _change_type} value of an update's post-image row. */ + String CHANGE_TYPE_UPDATE_POSTIMAGE = "update_postimage"; + /** A name to identify this changelog. */ String name(); diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala index 0ba801e3d6b7b..c736c3a8c0ef0 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala @@ -445,6 +445,7 @@ class Analyzer( new ResolveCatalogs(catalogManager) :: ResolveInsertInto :: ResolveRelations :: + ResolveChangelogTable :: ResolvePartitionSpec :: ResolveFieldNameAndPosition :: AddMetadataColumns :: diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala new file mode 100644 index 0000000000000..bdf9b9fed09cc --- /dev/null +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala @@ -0,0 +1,312 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.catalyst.analysis + +import org.apache.spark.sql.catalyst.expressions._ +import org.apache.spark.sql.catalyst.expressions.aggregate.{Count, Max, Min} +import org.apache.spark.sql.catalyst.plans.logical._ +import org.apache.spark.sql.catalyst.rules.Rule +import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 +import org.apache.spark.sql.connector.catalog.{Changelog, ChangelogInfo} +import org.apache.spark.sql.errors.QueryCompilationErrors +import org.apache.spark.sql.execution.datasources.v2.{ChangelogTable, DataSourceV2Relation} +import org.apache.spark.sql.types.{IntegerType, StringType} + +/** + * Post-processes a resolved [[ChangelogTable]] read to apply CDC option semantics + * (carry-over removal, update detection) and to enforce supported option combinations. + * + * Fires after [[ResolveRelations]] has wrapped the connector's [[Changelog]] in a + * [[ChangelogTable]]. Both batch ([[DataSourceV2Relation]]) and streaming + * ([[StreamingRelationV2]]) reads are handled: + * - Batch: the requested post-processing passes are injected as logical operators on top + * of the relation. Carry-over removal and update detection are fused into a single + * pass over a (rowId, _commit_version)-partitioned Window: the Filter drops CoW + * carry-over pairs (same rowVersion on both sides) and the subsequent Project relabels + * real delete+insert pairs as update_preimage / update_postimage. + * - Streaming: post-processing is not yet supported. If the requested options would + * require any post-processing, the rule throws an explicit [[AnalysisException]] to + * prevent silent wrong results. Streams that don't require post-processing pass + * through unchanged. + * + * Net change computation (`deduplicationMode = netChanges`) is not yet implemented and + * is rejected up-front for both batch and streaming. + */ +object ResolveChangelogTable extends Rule[LogicalPlan] { + + /** + * Reserved (`__spark_cdc_*`) column names used internally by post-processing; + * connectors must not emit columns with these names. + */ + object HelperColumn { + final val DelCnt = "__spark_cdc_del_cnt" + final val InsCnt = "__spark_cdc_ins_cnt" + final val MinRv = "__spark_cdc_min_rv" + final val MaxRv = "__spark_cdc_max_rv" + final val RvCnt = "__spark_cdc_rv_cnt" + + val all: Set[String] = Set(DelCnt, InsCnt, MinRv, MaxRv, RvCnt) + } + + override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp { + case rel @ DataSourceV2Relation(table: ChangelogTable, _, _, _, _, _) if !table.resolved => + val changelog = table.changelog + val req = evaluateRequirements(changelog, table.changelogInfo) + + val resolvedRel = rel.copy(table = table.copy(resolved = true)) + var updatedRel: LogicalPlan = resolvedRel + if (req.requiresCarryOverRemoval || req.requiresUpdateDetection) { + updatedRel = addRowLevelPostProcessing( + resolvedRel, changelog, req.requiresCarryOverRemoval, req.requiresUpdateDetection) + } + if (req.requiresNetChanges) { + updatedRel = injectNetChangeComputation(updatedRel, changelog) + } + updatedRel + + case rel @ StreamingRelationV2(_, _, table: ChangelogTable, _, _, _, _, _, _) + if !table.resolved => + // Streaming CDC reads do not yet apply post-processing. Run the same option / + // capability validation as the batch path so silent wrong results are impossible: + // either no post-processing would be required (fall through, return raw stream), + // or we throw an explicit AnalysisException. + val changelog = table.changelog + val req = evaluateRequirements(changelog, table.changelogInfo) + if (req.needsAny) { + throw QueryCompilationErrors.cdcStreamingPostProcessingNotSupported(changelog.name()) + } + rel.copy(table = table.copy(resolved = true)) + } + + // --------------------------------------------------------------------------- + // Option validation & Requirement Computation + // --------------------------------------------------------------------------- + + /** + * Captures which post-processing passes a CDC query requires, derived from the + * user-provided [[ChangelogInfo]] options and the connector-declared [[Changelog]] + * capability flags. + */ + private case class PostProcessingRequirements( + requiresCarryOverRemoval: Boolean, + requiresUpdateDetection: Boolean, + requiresNetChanges: Boolean) { + def needsAny: Boolean = + requiresCarryOverRemoval || requiresUpdateDetection || requiresNetChanges + } + + /** + * Validates CDC option/capability combinations and computes which post-processing + * passes are required. Throws an [[org.apache.spark.sql.AnalysisException]] for + * unsupported or contradictory combinations (currently: `netChanges` deduplication, + * and `computeUpdates` with surfaced carry-overs but no carry-over removal). + */ + private def evaluateRequirements( + changelog: Changelog, + options: ChangelogInfo): PostProcessingRequirements = { + // Net change computation is not yet implemented. + if (options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NET_CHANGES) { + throw QueryCompilationErrors.cdcNetChangesNotYetSupported(changelog.name()) + } + + val requiresCarryOverRemoval = + options.deduplicationMode() != ChangelogInfo.DeduplicationMode.NONE && + changelog.containsCarryoverRows() + val requiresUpdateDetection = + options.computeUpdates() && changelog.representsUpdateAsDeleteAndInsert() + val requiresNetChanges = + options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NET_CHANGES && + changelog.containsIntermediateChanges() + + // If carry-overs are surfaced and update detection is enabled without carry-over + // removal, carry-overs would be falsely classified as updates, leading to wrong + // results. Hence we throw. + if (requiresUpdateDetection && + changelog.containsCarryoverRows() && + options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NONE) { + throw QueryCompilationErrors.cdcUpdateDetectionRequiresCarryOverRemoval( + changelog.name()) + } + + PostProcessingRequirements( + requiresCarryOverRemoval, requiresUpdateDetection, requiresNetChanges) + } + + // --------------------------------------------------------------------------- + // Row Level Post Processing (Update Detection & Carry-over Removal) + // --------------------------------------------------------------------------- + + /** + * Adds row-level post-processing (carry-over removal and/or update detection) on top of + * the given plan. `counts` = per-partition delete and insert change_type row counts over + * `(rowId, _commit_version)`. `rv bounds` = per-partition min/max of `rowVersion`. + * Equal bounds signal a copy-on-write carry-over. + * - both active -> Window(counts + rv bounds) -> Filter -> Project(relabel) -> Drop helpers + * - carry-over only -> Window(counts + rv bounds) -> Filter -> Drop helpers + * - update only -> Window(counts only) -> Project(relabel) -> Drop helpers + * - neither -> not invoked (caller guards this case) + */ + private def addRowLevelPostProcessing( + plan: LogicalPlan, + cl: Changelog, + requiresCarryOverRemoval: Boolean, + requiresUpdateDetection: Boolean): LogicalPlan = { + // Row-version bounds in the Window are needed iff we filter carry-over pairs. + var modifiedPlan = addPostProcessingWindow(plan, cl, + includeRowVersionBounds = requiresCarryOverRemoval) + if (requiresCarryOverRemoval) modifiedPlan = addCarryOverPairFilter(modifiedPlan) + if (requiresUpdateDetection) modifiedPlan = addUpdateRelabelProjection(modifiedPlan) + removeHelperColumns(modifiedPlan) + } + + /** + * Adds a Window node partitioned by (rowId, _commit_version) that computes + * `_del_cnt` and `_ins_cnt` per partition, and, when `includeRowVersionBounds` + * is true, additionally `_min_rv` / `_max_rv` / `_rv_cnt` (min, max and non-null + * count of `Changelog.rowVersion()`). + * + * `_del_cnt` / `_ins_cnt` drive update detection (1 each -> relabel as + * update_preimage / update_postimage). `_min_rv` / `_max_rv` / `_rv_cnt` drive + * carry-over detection (within a delete+insert pair, `_rv_cnt = 2` AND equal + * bounds signal a CoW carry-over). + */ + private def addPostProcessingWindow( + plan: LogicalPlan, + cl: Changelog, + includeRowVersionBounds: Boolean): LogicalPlan = { + val changeTypeAttr = getAttribute(plan, "_change_type") + val rowIdExprs = V2ExpressionUtils.resolveRefs[NamedExpression](cl.rowId().toSeq, plan) + val commitVersionAttr = getAttribute(plan, "_commit_version") + val partitionByCols = rowIdExprs ++ Seq(commitVersionAttr) + val windowSpec = WindowSpecDefinition(partitionByCols, Nil, UnspecifiedFrame) + + val insertIf = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_INSERT)), + Literal(1), Literal(null, IntegerType)) + val deleteIf = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_DELETE)), + Literal(1), Literal(null, IntegerType)) + + val insCntAlias = Alias(WindowExpression( + Count(Seq(insertIf)).toAggregateExpression(), windowSpec), HelperColumn.InsCnt)() + val delCntAlias = Alias(WindowExpression( + Count(Seq(deleteIf)).toAggregateExpression(), windowSpec), HelperColumn.DelCnt)() + val baseAliases = Seq(delCntAlias, insCntAlias) + val rowVersionAliases = if (includeRowVersionBounds) { + val rowVersionExpr = + V2ExpressionUtils.resolveRef[NamedExpression](cl.rowVersion(), plan) + Seq( + Alias(WindowExpression( + Min(rowVersionExpr).toAggregateExpression(), windowSpec), HelperColumn.MinRv)(), + Alias(WindowExpression( + Max(rowVersionExpr).toAggregateExpression(), windowSpec), HelperColumn.MaxRv)(), + Alias(WindowExpression( + Count(Seq(rowVersionExpr)).toAggregateExpression(), windowSpec), HelperColumn.RvCnt)()) + } else { + Seq.empty + } + Window(baseAliases ++ rowVersionAliases, partitionByCols, Nil, plan) + } + + /** + * Adds a Filter node that drops rows belonging to a CoW carry-over pair. + * A pair is a carry-over iff + * `_del_cnt = 1 AND _ins_cnt = 1 AND _rv_cnt = 2 AND _min_rv = _max_rv`. + * The `_rv_cnt = 2` clause guards against a NULL rowVersion silently matching + * `_min_rv = _max_rv` (Spark's min/max skip NULLs). + */ + private def addCarryOverPairFilter(input: LogicalPlan): LogicalPlan = { + val delCnt = getAttribute(input, HelperColumn.DelCnt) + val insCnt = getAttribute(input, HelperColumn.InsCnt) + val minRv = getAttribute(input, HelperColumn.MinRv) + val maxRv = getAttribute(input, HelperColumn.MaxRv) + val rvCnt = getAttribute(input, HelperColumn.RvCnt) + + val isCarryoverPair = And( + And(EqualTo(delCnt, Literal(1L)), EqualTo(insCnt, Literal(1L))), + And(EqualTo(rvCnt, Literal(2L)), EqualTo(minRv, maxRv))) + Filter(Not(isCarryoverPair), input) + } + + /** + * Adds a Project node that rewrites `_change_type` to `update_preimage` / + * `update_postimage` whenever a delete+insert pair is present in the partition. + * Expects the input to expose `_del_cnt` and `_ins_cnt`. + */ + private def addUpdateRelabelProjection(input: LogicalPlan): LogicalPlan = { + val changeTypeAttr = getAttribute(input, "_change_type") + val delCnt = getAttribute(input, HelperColumn.DelCnt) + val insCnt = getAttribute(input, HelperColumn.InsCnt) + + val isUpdate = And( + EqualTo(delCnt, Literal(1L)), + EqualTo(insCnt, Literal(1L))) + val isInvalid = Or(GreaterThan(delCnt, Literal(1L)), GreaterThan(insCnt, Literal(1L))) + val updateType = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_INSERT)), + Literal(Changelog.CHANGE_TYPE_UPDATE_POSTIMAGE), + Literal(Changelog.CHANGE_TYPE_UPDATE_PREIMAGE)) + + val raiseInvalid = RaiseError( + Literal("CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION"), + CreateMap(Nil), + StringType) + val caseExpr = CaseWhen(Seq(isInvalid -> raiseInvalid, isUpdate -> updateType), changeTypeAttr) + + val projectList = input.output.map { attr => + if (attr.name == "_change_type") Alias(caseExpr, "_change_type")() + else attr + } + Project(projectList, input) + } + + // --------------------------------------------------------------------------- + // Net Change Computation + // --------------------------------------------------------------------------- + + /** + * Collapses multiple changes per row identity into the net effect. + * Not yet implemented. + */ + private def injectNetChangeComputation( + plan: LogicalPlan, + cl: Changelog): LogicalPlan = { + plan + } + + // --------------------------------------------------------------------------- + // Helpers + // --------------------------------------------------------------------------- + + /** + * Removes any helper columns (see [[HelperColumn]]) that earlier steps added to the + * plan. Helper columns not present in the input are silently ignored, so this method + * can be applied unconditionally regardless of which post-processing steps ran. + */ + private def removeHelperColumns(input: LogicalPlan): LogicalPlan = { + Project(input.output.filterNot(a => HelperColumn.all.contains(a.name)), input) + } + + /** + * Looks up an attribute by name in a plan's output. Throws a clear error if missing -- + * used for required columns like `_change_type` / `_commit_version` / helper columns + * added by earlier steps; a missing column is always a programming error. + */ + private def getAttribute(plan: LogicalPlan, name: String): Attribute = + plan.output.find(_.name == name).getOrElse( + throw new IllegalStateException( + s"Required column '$name' not found in plan output: " + + plan.output.map(_.name).mkString(", "))) +} diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala index bd1e876c9fbd6..512d6f6305266 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala @@ -3862,6 +3862,25 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat messageParameters = Map("catalogName" -> catalogName)) } + def cdcUpdateDetectionRequiresCarryOverRemoval( + changelogName: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CDC_OPTION.UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL", + messageParameters = Map("changelogName" -> changelogName)) + } + + def cdcNetChangesNotYetSupported(changelogName: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", + messageParameters = Map("changelogName" -> changelogName)) + } + + def cdcStreamingPostProcessingNotSupported(changelogName: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", + messageParameters = Map("changelogName" -> changelogName)) + } + def invalidCdcOptionConflictingRangeTypes(): Throwable = { new AnalysisException( errorClass = "INVALID_CDC_OPTION.CONFLICTING_RANGE_TYPES", diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala index 8521df3db2ff0..bb5a03f64990d 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala @@ -33,7 +33,8 @@ import org.apache.spark.sql.util.CaseInsensitiveStringMap */ case class ChangelogTable( changelog: Changelog, - changelogInfo: ChangelogInfo) extends Table with SupportsRead { + changelogInfo: ChangelogInfo, + resolved: Boolean = false) extends Table with SupportsRead { override def name: String = changelog.name diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala index c47ed2668e3b4..3a37b0a84fa26 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala @@ -23,6 +23,7 @@ import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.analysis.NoSuchTableException import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.ChangelogRange.{TimestampRange, UnboundedRange, VersionRange} +import org.apache.spark.sql.connector.expressions.{FieldReference, NamedReference} import org.apache.spark.sql.connector.read._ import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, Offset} import org.apache.spark.sql.types._ @@ -44,6 +45,22 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { private var _lastChangelogInfo: Option[ChangelogInfo] = None def lastChangelogInfo: Option[ChangelogInfo] = _lastChangelogInfo + // Per-table overrides for Changelog properties (carry-over rows, intermediate changes, + // update representation, row identity). Tests can set these to exercise post-processing. + private val changelogProperties: mutable.Map[String, ChangelogProperties] = + mutable.Map.empty + + /** + * Override the [[Changelog]] properties returned for a given table. + * Defaults are: containsCarryoverRows=false, containsIntermediateChanges=false, + * representsUpdateAsDeleteAndInsert=false, no rowId, no rowVersion. + */ + def setChangelogProperties( + ident: Identifier, + properties: ChangelogProperties): Unit = { + changelogProperties(ident.toString) = properties + } + override def loadChangelog( ident: Identifier, changelogInfo: ChangelogInfo): Changelog = { @@ -58,8 +75,9 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { // _commit_version is at index numDataCols + 1 (after _change_type) val commitVersionIdx = numDataCols + 1 val filtered = filterByRange(allRows.toSeq, commitVersionIdx, changelogInfo.range()) + val props = changelogProperties.getOrElse(ident.toString, ChangelogProperties()) new InMemoryChangelog( - table.name + "_changelog", table.columns, filtered) + table.name + "_changelog", table.columns, filtered, props) } /** @@ -109,15 +127,42 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { } } +/** + * Configurable properties for [[InMemoryChangelog]] that test cases can use to exercise + * Spark's post-processing (carry-over removal, update detection, net changes). + * + * @param containsCarryoverRows whether the change stream may contain identical CoW pairs + * @param containsIntermediateChanges whether multiple changes per row may exist + * @param representsUpdateAsDeleteAndInsert whether updates appear as raw delete+insert + * @param rowIdNames optional row identity columns as top-level names (e.g. Seq("id")) + * @param rowIdPaths optional row identity paths for nested struct fields + * (e.g. Seq(Seq("payload", "id"))); takes precedence over rowIdNames + * @param rowVersionName optional row version column (e.g. Some("row_commit_version")); + * must be a per-row version that distinguishes carry-overs from + * real updates. Do NOT pass the commit version, which is constant + * within a partition and would cause every delete+insert pair to + * look like a carry-over + */ +case class ChangelogProperties( + containsCarryoverRows: Boolean = false, + containsIntermediateChanges: Boolean = false, + representsUpdateAsDeleteAndInsert: Boolean = false, + rowIdNames: Seq[String] = Seq.empty, + rowIdPaths: Seq[Seq[String]] = Seq.empty, + rowVersionName: Option[String] = None) + /** * A test [[Changelog]] that returns pre-populated change rows. * - * Reports `containsCarryoverRows = false` so Spark skips carry-over removal. + * Properties (carry-over presence, update representation, row identity) are configurable + * via the [[ChangelogProperties]] parameter so tests can exercise different code paths + * in Spark's post-processing analyzer rule. */ class InMemoryChangelog( tableName: String, dataColumns: Array[Column], - changeRows: Seq[InternalRow]) extends Changelog { + changeRows: Seq[InternalRow], + properties: ChangelogProperties = ChangelogProperties()) extends Changelog { private val cdcColumns: Array[Column] = dataColumns ++ Array( Column.create("_change_type", StringType), @@ -128,11 +173,25 @@ class InMemoryChangelog( override def columns(): Array[Column] = cdcColumns - override def containsCarryoverRows(): Boolean = false + override def containsCarryoverRows(): Boolean = properties.containsCarryoverRows + + override def containsIntermediateChanges(): Boolean = properties.containsIntermediateChanges - override def containsIntermediateChanges(): Boolean = false + override def representsUpdateAsDeleteAndInsert(): Boolean = + properties.representsUpdateAsDeleteAndInsert - override def representsUpdateAsDeleteAndInsert(): Boolean = false + override def rowId(): Array[NamedReference] = { + if (properties.rowIdPaths.nonEmpty) { + properties.rowIdPaths.map(parts => FieldReference(parts): NamedReference).toArray + } else { + properties.rowIdNames.map(name => FieldReference.column(name): NamedReference).toArray + } + } + + override def rowVersion(): NamedReference = properties.rowVersionName match { + case Some(name) => FieldReference.column(name) + case None => super.rowVersion() + } override def newScanBuilder( options: CaseInsensitiveStringMap): ScanBuilder = { diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala index 006b645193023..9622d23122318 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala @@ -418,27 +418,22 @@ class ChangelogEndToEndSuite extends SharedSparkSession { ChangelogInfo.DeduplicationMode.NONE) } - test("changes() passes deduplicationMode and computeUpdates to catalog") { + test("changes() passes computeUpdates to catalog") { catalog.addChangeRows(ident, Seq( makeChangeRow(1L, "a", "insert", 1L, 1000000L))) // DataFrame API spark.read .option("startingVersion", "1") - .option("deduplicationMode", "netChanges") .option("computeUpdates", "true") .changes(fullTableName) .collect() - val info1 = catalog.lastChangelogInfo.get - assert(info1.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) - assert(info1.computeUpdates() === true) + assert(catalog.lastChangelogInfo.get.computeUpdates() === true) // SQL sql(s"SELECT * FROM $fullTableName CHANGES FROM VERSION 1 " + - "WITH (deduplicationMode = 'netChanges', computeUpdates = 'true')").collect() - val info2 = catalog.lastChangelogInfo.get - assert(info2.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) - assert(info2.computeUpdates() === true) + "WITH (computeUpdates = 'true')").collect() + assert(catalog.lastChangelogInfo.get.computeUpdates() === true) } // ---------- Batch: timestamp range ---------- @@ -589,23 +584,20 @@ class ChangelogEndToEndSuite extends SharedSparkSession { // ---------- Streaming: CDC options ---------- - test("streaming changes() passes deduplicationMode and computeUpdates to catalog") { + test("streaming changes() passes computeUpdates to catalog") { catalog.addChangeRows(ident, Seq( makeChangeRow(1L, "a", "insert", 1L, 1000000L))) // DataFrame API val dfApiStream = spark.readStream .option("startingVersion", "1") - .option("deduplicationMode", "netChanges") .option("computeUpdates", "true") .changes(fullTableName) val q1 = dfApiStream.writeStream .format("memory").queryName("cdc_stream_opts_df").start() try { q1.processAllAvailable() - val info1 = catalog.lastChangelogInfo.get - assert(info1.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) - assert(info1.computeUpdates() === true) + assert(catalog.lastChangelogInfo.get.computeUpdates() === true) } finally { q1.stop() } @@ -613,14 +605,12 @@ class ChangelogEndToEndSuite extends SharedSparkSession { // SQL val sqlStream = sql( s"SELECT * FROM STREAM $fullTableName CHANGES FROM VERSION 1 " + - "WITH (deduplicationMode = 'netChanges', computeUpdates = 'true')") + "WITH (computeUpdates = 'true')") val q2 = sqlStream.writeStream .format("memory").queryName("cdc_stream_opts_sql").start() try { q2.processAllAvailable() - val info2 = catalog.lastChangelogInfo.get - assert(info2.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) - assert(info2.computeUpdates() === true) + assert(catalog.lastChangelogInfo.get.computeUpdates() === true) } finally { q2.stop() } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala index db6817b0c212c..d403db1e62bf9 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala @@ -17,13 +17,14 @@ package org.apache.spark.sql.connector -import java.util +import java.util.Collections import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 import org.apache.spark.sql.connector.catalog._ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.ChangelogRange +import org.apache.spark.sql.connector.expressions.Transform import org.apache.spark.sql.execution.datasources.v2.{ChangelogTable, DataSourceV2Relation} import org.apache.spark.sql.test.SharedSparkSession import org.apache.spark.sql.types.{LongType, StringType} @@ -63,8 +64,8 @@ class ChangelogResolutionSuite extends SharedSparkSession { Array( Column.create("id", LongType), Column.create("data", StringType)), - Array.empty, - new util.HashMap[String, String]()) + Array.empty[Transform], + Collections.emptyMap[String, String]()) val noCdcCat = spark.sessionState.catalogManager.catalog(noCdcCatalogName).asTableCatalog val ident2 = Identifier.of(Array.empty, "test_table") @@ -76,8 +77,8 @@ class ChangelogResolutionSuite extends SharedSparkSession { Array( Column.create("id", LongType), Column.create("data", StringType)), - Array.empty, - new util.HashMap[String, String]()) + Array.empty[Transform], + Collections.emptyMap[String, String]()) } test("CHANGES clause resolves to DataSourceV2Relation with ChangelogTable") { @@ -203,4 +204,69 @@ class ChangelogResolutionSuite extends SharedSparkSession { assert(range.startingVersion() == "1") assert(range.endingVersion().get() == "5") } + + // =========================================================================== + // Streaming post-processing rejection + // =========================================================================== + // + // Streaming CDC reads bypass the post-processing analyzer rule's transformation + // path. To prevent silent wrong results when the requested options would require + // post-processing, the rule throws an explicit AnalysisException for streaming. + + /** Re-creates the test table with non-nullable columns suitable as rowId / rowVersion. */ + private def recreatePostProcessingTable(): Identifier = { + val cat = spark.sessionState.catalogManager.catalog(cdcCatalogName).asTableCatalog + val ident = Identifier.of(Array.empty, "test_table") + if (cat.tableExists(ident)) cat.dropTable(ident) + cat.createTable( + ident, + Array( + Column.create("id", LongType, false), + Column.create("row_commit_version", LongType, false)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + ident + } + + test("DataStreamReader - changes() with carry-over capability throws") { + val ident = recreatePostProcessingTable() + val cat = spark.sessionState.catalogManager + .catalog(cdcCatalogName) + .asInstanceOf[InMemoryChangelogCatalog] + cat.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + spark.readStream + .changes(s"$cdcCatalogName.test_table") + .queryExecution.analyzed + }, + condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", + parameters = Map("changelogName" -> s"$cdcCatalogName.test_table_changelog")) + } + + test("DataStreamReader - changes() with computeUpdates throws") { + val ident = recreatePostProcessingTable() + val cat = spark.sessionState.catalogManager + .catalog(cdcCatalogName) + .asInstanceOf[InMemoryChangelogCatalog] + cat.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + spark.readStream + .option("computeUpdates", "true") + .option("deduplicationMode", "none") + .changes(s"$cdcCatalogName.test_table") + .queryExecution.analyzed + }, + condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", + parameters = Map("changelogName" -> s"$cdcCatalogName.test_table_changelog")) + } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala new file mode 100644 index 0000000000000..353472a035f91 --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala @@ -0,0 +1,1034 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector + +import java.util.Collections + +import org.scalatest.BeforeAndAfterEach + +import org.apache.spark.SparkRuntimeException +import org.apache.spark.sql.{AnalysisException, QueryTest, Row} +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 +import org.apache.spark.sql.connector.catalog.{ + ChangelogProperties, Column, Identifier, InMemoryChangelogCatalog} +import org.apache.spark.sql.connector.catalog.Changelog.{ + CHANGE_TYPE_DELETE, CHANGE_TYPE_INSERT, CHANGE_TYPE_UPDATE_POSTIMAGE, + CHANGE_TYPE_UPDATE_PREIMAGE} +import org.apache.spark.sql.connector.expressions.Transform +import org.apache.spark.sql.execution.datasources.v2.ChangelogTable +import org.apache.spark.sql.test.SharedSparkSession +import org.apache.spark.sql.types.{ + BinaryType, BooleanType, DoubleType, LongType, StringType, StructField, StructType} +import org.apache.spark.unsafe.types.UTF8String + +/** + * Tests for [[org.apache.spark.sql.catalyst.analysis.ResolveChangelogTable]] using the + * in-memory changelog catalog. These tests don't depend on Delta or any specific connector; + * they directly control what the connector "returns" by populating the in-memory changelog + * with hand-crafted change rows. + * + * Each test sets up [[ChangelogProperties]] on the catalog to enable specific post-processing + * paths (carry-over removal, update detection) and then verifies that Spark's analyzer rule + * correctly transforms the plan and produces the expected output. + */ +class ResolveChangelogTablePostProcessingSuite + extends QueryTest + with SharedSparkSession + with BeforeAndAfterEach { + + private val catalogName = "cdc_test_catalog" + private val testTableName = "events" + + override def beforeAll(): Unit = { + super.beforeAll() + spark.conf.set( + s"spark.sql.catalog.$catalogName", + classOf[InMemoryChangelogCatalog].getName) + } + + override def beforeEach(): Unit = { + super.beforeEach() + val cat = catalog + val ident = Identifier.of(Array.empty, testTableName) + if (cat.tableExists(ident)) cat.dropTable(ident) + cat.clearChangeRows(ident) + cat.setChangelogProperties(ident, ChangelogProperties()) + cat.createTable( + ident, + Array( + Column.create("id", LongType), + Column.create("name", StringType), + Column.create("row_commit_version", LongType, false)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + } + + private def catalog: InMemoryChangelogCatalog = { + spark.sessionState.catalogManager + .catalog(catalogName) + .asInstanceOf[InMemoryChangelogCatalog] + } + + private def ident = Identifier.of(Array.empty, testTableName) + + /** + * Helper to create a change row matching schema + * (id, name, row_commit_version, _change_type, _commit_version, _commit_timestamp). + * + * `rowCommitVersion` follows Delta row-tracking semantics: carry-over pairs (CoW-rewritten + * unchanged rows) share the same value on both sides; real updates carry the OLD value on + * the delete side and the NEW value on the insert side. Defaults to `commitVersion` for + * tests that don't exercise carry-over removal. + */ + private def changeRow( + id: Long, + name: String, + changeType: String, + commitVersion: Long, + rowCommitVersion: Long = -1L, + commitTimestamp: Long = 0L): InternalRow = { + val rcv = if (rowCommitVersion == -1L) commitVersion else rowCommitVersion + InternalRow( + id, + UTF8String.fromString(name), + rcv, + UTF8String.fromString(changeType), + commitVersion, + commitTimestamp) + } + + // =========================================================================== + // Carry-Over Removal + // =========================================================================== + + test("carry-over removal drops identical delete+insert pairs") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // v1: insert Alice and Bob (rcv=1 each) + // v2: real delete Alice (preimage carries old rcv=1); + // carry-over for Bob (CoW, rcv unchanged on both sides) + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // carry-over + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) // carry-over (same rcv) + + checkAnswer( + sql( + s"SELECT id, name, _change_type, _commit_version " + + s"FROM $catalogName.$testTableName CHANGES FROM VERSION 1 TO VERSION 2"), + Seq( + Row(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + Row(2L, "Bob", CHANGE_TYPE_INSERT, 1L), + Row(1L, "Alice", CHANGE_TYPE_DELETE, 2L))) + } + + test("deduplicationMode=none keeps all carry-over rows") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) + + checkAnswer( + sql( + s"SELECT id FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (deduplicationMode = 'none')"), + Seq(Row(1L), Row(2L), Row(2L))) + } + + test("NULL rowVersion on one side is NOT silently dropped as carry-over") { + // Regression for a NULL-safety hole: min/max skip NULLs, so _min_rv = _max_rv alone + // would match a pair with one NULL and one non-null rowVersion. The _rv_cnt = 2 + // clause in the carry-over filter prevents that. + // + // The fixture table here declares `row_commit_version` as nullable so the optimizer + // is not allowed to fold IsNull(non-nullable-col) to false; the NULL is a legitimate + // value the guard must defend against. + val nullableRcvTable = "events_nullable_rcv" + val nullableIdent = Identifier.of(Array.empty, nullableRcvTable) + val cat = catalog + if (cat.tableExists(nullableIdent)) cat.dropTable(nullableIdent) + cat.clearChangeRows(nullableIdent) + cat.createTable( + nullableIdent, + Array( + Column.create("id", LongType), + Column.create("name", StringType), + Column.create("row_commit_version", LongType, true)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + cat.setChangelogProperties(nullableIdent, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + cat.addChangeRows(nullableIdent, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2: one side has NULL rowVersion (buggy connector), the other has a real value. + InternalRow(1L, UTF8String.fromString("Alice"), null, + UTF8String.fromString(CHANGE_TYPE_DELETE), 2L, 0L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 5L))) + + checkAnswer( + sql(s"SELECT id, name, _change_type, _commit_version " + + s"FROM $catalogName.$nullableRcvTable CHANGES FROM VERSION 1 TO VERSION 2"), + Seq( + Row(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + Row(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + Row(1L, "Alice", CHANGE_TYPE_INSERT, 2L))) + } + + // =========================================================================== + // Update Detection + // =========================================================================== + + test("update detection relabels delete+insert with different data as update") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = false, // no carry-overs in this test + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + // v2: Alice -> Robert (delete old, insert new) + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Robert", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version " + + s"FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + + assert(descs.contains("1:Alice:insert"), s"v1 insert. Got: ${descs.mkString(",")}") + assert(descs.contains("1:Alice:update_preimage")) + assert(descs.contains("1:Robert:update_postimage")) + // No raw delete/insert at v2 + assert(!descs.contains("1:Alice:delete")) + assert(!descs.contains("1:Robert:insert")) + } + + test("delete and insert in different versions are NOT labeled as update") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 3L))) + + val rows = sql( + s"SELECT _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 3 " + + s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") + .collect() + + assert(!rows.exists(_.getString(0).contains("update_")), + "Delete and insert in different versions should not be labeled as update") + } + + // =========================================================================== + // Composite rowId: partitioning uses every rowId column + // =========================================================================== + // + // With a composite rowId such as Seq("id", "name"), the (rowId, _commit_version) + // window partition must include BOTH columns. A regression that drops one of the + // rowId columns would either falsely merge two different row identities into one + // partition (silently mislabeling unrelated delete/insert pairs as updates) or + // trip the UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION runtime guard. + + test("update detection with composite rowId keeps different (id, name) tuples raw") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id", "name"), + rowVersionName = Some("row_commit_version"))) + + // delete (1, Alice) and insert (1, Bob) at v2. These are DIFFERENT composite + // rowIds; they must NOT be relabeled as update. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}").toSet + + assert(descs == Set("1:Alice:delete", "1:Bob:insert"), + s"Composite rowId must keep different (id, name) tuples raw. Got: $descs") + } + + test("carry-over removal with composite rowId removes pairs per (id, name) tuple") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id", "name"), + rowVersionName = Some("row_commit_version"))) + + // Two independent carry-over pairs at v2, both with id=1 but different names. + // With correct composite-rowId partitioning, each pair lives in its own + // (id, name, _commit_version) partition, has _del_cnt=1 / _ins_cnt=1 and equal + // _min_rv / _max_rv, and gets dropped. With broken (id-only) partitioning, the + // four rows would collapse into one partition with _del_cnt=2 / _ins_cnt=2 and + // the carry-over filter (which requires =1) would keep them all. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), + changeRow(1L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version " + + s"FROM $catalogName.$testTableName CHANGES FROM VERSION 2 TO VERSION 2") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + assert(rows.isEmpty, + s"Both Alice and Bob carry-over pairs at v2 should be removed. Got: ${descs.mkString(",")}") + } + + // =========================================================================== + // No row identity: post-processing skipped + // =========================================================================== + + test("no capability flags -> post-processing not injected in plan") { + // Default ChangelogProperties has no capability flags set; the rule sees nothing to do. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L))) + + val df = sql( + s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + + val plan = df.queryExecution.analyzed.treeString + assert(!plan.contains("__spark_cdc_del_cnt"), + s"Plan must not contain post-processing window helpers. Plan:\n$plan") + assert(!plan.contains("__spark_cdc_ins_cnt"), + s"Plan must not contain post-processing window helpers. Plan:\n$plan") + } + + test("streaming without post-processing options passes through") { + // Streaming reads with no capability flags on the connector and no + // post-processing options must resolve without the rule throwing. + val df = spark.readStream + .option("startingVersion", "1") + .changes(s"$catalogName.$testTableName") + val analyzed = df.queryExecution.analyzed + val plan = analyzed.treeString + assert(!plan.contains("__spark_cdc_del_cnt"), + s"Streaming plan must not contain post-processing helpers. Plan:\n$plan") + + // Positive assertion: the rule actually fired on the streaming relation. Without this, + // a regression that deletes the streaming arm of `ResolveChangelogTable.apply` would + // also pass the absence-of-helpers check above. + val tableResolved = analyzed.collectFirst { + case rel: StreamingRelationV2 if rel.table.isInstanceOf[ChangelogTable] => + rel.table.asInstanceOf[ChangelogTable].resolved + } + assert(tableResolved.contains(true), + s"Expected ChangelogTable to be marked resolved by the rule. Plan:\n$plan") + } + + test("streaming with post-processing options is rejected") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + exception = intercept[AnalysisException] { + spark.readStream + .option("startingVersion", "1") + .changes(s"$catalogName.$testTableName") + .queryExecution.analyzed + }, + condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", + parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) + } + + // =========================================================================== + // Combined + // =========================================================================== + + test("carry-over removal and update detection combined") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // v1: insert Alice (rcv=1), Bob (rcv=1) + // v2: Alice carry-over (CoW, rcv unchanged), Bob real update (old rcv=1, new rcv=2) + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // carry-over + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), // carry-over + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // update preimage + changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) // update postimage + + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}").toSet + + // v1 inserts + assert(descs.contains("1:Alice:insert")) + assert(descs.contains("2:Bob:insert")) + // Alice carry-over dropped + assert(!descs.contains("1:Alice:delete")) + // Bob -> Robert as update + assert(descs.contains("2:Bob:update_preimage")) + assert(descs.contains("2:Robert:update_postimage")) + // Should be exactly 4 rows + assert(rows.length == 4, s"Expected 4 rows, got ${rows.length}: ${descs.mkString(",")}") + } + + // =========================================================================== + // computeUpdates default (false) keeps raw delete+insert + // =========================================================================== + + test("without computeUpdates, delete+insert with different data stays raw") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // Alice: carry-over (CoW, rcv unchanged on both sides) + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), + // Bob -> Robert: real change (old rcv on pre, new rcv on post) + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) + + // Default computeUpdates=false: do NOT relabel, but DO drop carry-overs + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + + assert(descs.contains("2:Bob:delete"), s"Bob delete remains raw. Got: ${descs.mkString(",")}") + assert(descs.contains("2:Robert:insert"), "Robert insert remains raw") + assert(!descs.exists(_.contains("update_")), "No update_* without computeUpdates") + assert(!descs.contains("1:Alice:delete"), "Alice carry-over removed") + } + + test("update detection on pure inserts leaves them as inserts") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .collect() + + assert(rows.length == 2) + assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT), + s"Pure inserts must stay 'insert'. Got: ${rows.map(_.getString(1)).mkString(",")}") + } + + // =========================================================================== + // Keep Carry-over Rows and deduplication flag tests + // =========================================================================== + + test("computeUpdates with deduplicationMode=none is rejected on COW connector") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 " + + s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") + }, + condition = "INVALID_CDC_OPTION.UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL", + parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) + } + + test("computeUpdates with deduplicationMode=none is allowed on non-COW connector") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = false, // MOR-style: no carry-overs possible + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + // v2: Alice -> Robert (delete old, insert new) + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Robert", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 " + + s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + assert(descs.contains("1:Alice:update_preimage"), + s"Expected Alice update_preimage. Got: ${descs.mkString(",")}") + assert(descs.contains("1:Robert:update_postimage"), + s"Expected Robert update_postimage. Got: ${descs.mkString(",")}") + } + + // =========================================================================== + // Contract enforcement: at most one delete + one insert per (rowId, version) + // =========================================================================== + // + // With `representsUpdateAsDeleteAndInsert = true` and `containsIntermediateChanges = false`, + // the `Changelog` contract guarantees at most one logical change per (rowId, _commit_version) + // partition. The update-relabel projection enforces this at runtime: if it sees more than one + // delete or more than one insert in a partition, it raises + // INVALID_CDC_OPTION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION instead of silently + // mislabeling extra rows as updates. + + test("update detection raises on multiple inserts for same (rowId, _commit_version)") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // Contract violation: 2 inserts for id=1 at v2. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Alice2", CHANGE_TYPE_INSERT, 2L), + changeRow(1L, "Alice3", CHANGE_TYPE_INSERT, 2L))) + + checkError( + intercept[SparkRuntimeException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") + .collect() + }, + condition = "CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION", + parameters = Map.empty) + } + + test("update detection raises on multiple deletes for same (rowId, _commit_version)") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // Contract violation: 2 deletes for id=1 at v2. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Alice2", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Alice3", CHANGE_TYPE_INSERT, 2L))) + + checkError( + intercept[SparkRuntimeException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") + .collect() + }, + condition = "CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION", + parameters = Map.empty) + } + + // =========================================================================== + // Net changes deduplication: not yet supported + // =========================================================================== + // + // `deduplicationMode = netChanges` collapses multiple changes per row identity into the + // net effect. It is not yet implemented in [[ResolveChangelogTable]]. + + test("deduplicationMode=netChanges is rejected when connector emits intermediate changes") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsIntermediateChanges = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 " + + s"WITH (deduplicationMode = 'netChanges')") + }, + condition = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", + parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) + } + + test("deduplicationMode=netChanges is rejected even when connector has no intermediate changes") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsIntermediateChanges = false, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 " + + s"WITH (deduplicationMode = 'netChanges')") + }, + condition = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", + parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) + } + + // =========================================================================== + // Range edge cases + // =========================================================================== + + test("multiple operations across versions") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + // v1: insert 3 rows (rcv=1 each) + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2: delete Alice (preimage carries old rcv=1); CoW carry-overs for Bob/Charlie + // keep rcv=1 on both sides (row unchanged). + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), + // v3: update Bob -> Robert (old rcv=1, new rcv=3); CoW carry-over for Charlie (rcv=1) + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 3L, rowCommitVersion = 1L), + changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 3L, rowCommitVersion = 3L), + changeRow(3L, "Charlie", CHANGE_TYPE_DELETE, 3L, rowCommitVersion = 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L, rowCommitVersion = 1L), + // v4: insert Diana (rcv=4) + changeRow(4L, "Diana", CHANGE_TYPE_INSERT, 4L, rowCommitVersion = 4L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 4 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}").toSet + + // v1 + assert(descs.contains("1:Alice:insert:v1")) + assert(descs.contains("2:Bob:insert:v1")) + assert(descs.contains("3:Charlie:insert:v1")) + // v2 + assert(descs.contains("1:Alice:delete:v2")) + assert(!descs.contains("2:Bob:delete:v2"), "Bob carry-over dropped") + assert(!descs.contains("3:Charlie:delete:v2"), "Charlie carry-over dropped") + // v3 + assert(descs.contains("2:Bob:update_preimage:v3")) + assert(descs.contains("2:Robert:update_postimage:v3")) + assert(!descs.contains("3:Charlie:delete:v3"), "Charlie carry-over dropped in v3") + // v4 + assert(descs.contains("4:Diana:insert:v4")) + } + + test("larger insert batch returns all rows") { + catalog.addChangeRows(ident, (1 to 5).map(i => + changeRow(i.toLong, ('A' + i - 1).toChar.toString, CHANGE_TYPE_INSERT, 1L))) + + val rows = sql( + s"SELECT id, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 1 WITH (deduplicationMode = 'none')") + .collect() + + assert(rows.length == 5) + assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT)) + } + + test("DELETE all rows: no carry-over inserts at v2") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // v1 inserts carry rcv=1; v2 deletes carry the old rcv=1 (rcv tracks last modification) + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2") + .orderBy("_commit_version", "id") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") + + assert(descs.contains("1:Alice:insert:v1")) + assert(descs.contains("2:Bob:insert:v1")) + assert(descs.contains("1:Alice:delete:v2")) + assert(descs.contains("2:Bob:delete:v2")) + assert(!descs.exists(_.contains("insert:v2")), "No inserts at v2") + } + + test("UPDATE all rows: every row gets update_pre/postimage, no carry-overs") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // Every v2 row is a real update: delete side carries old rcv=1, insert side new rcv=2. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Alice_updated", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob_updated", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}").toSet + + assert(descs.contains("1:Alice:update_preimage:v2")) + assert(descs.contains("1:Alice_updated:update_postimage:v2")) + assert(descs.contains("2:Bob:update_preimage:v2")) + assert(descs.contains("2:Bob_updated:update_postimage:v2")) + assert(rows.length == 6, s"Expected 2 inserts + 2 pre + 2 post. Got ${rows.length}") + } + + test("append-only workload: all inserts, no carry-over needed") { + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) + + val rows = sql( + s"SELECT id, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 3") + .collect() + + assert(rows.length == 3) + assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT)) + } + + test("carry-over removal with many rows: only real change remains") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // 10 inserts at v1 (rcv=1 each). At v2: delete row 5; CoW writes 9 carry-over pairs + // (rcv unchanged since v1, i.e. rcv=1 on both sides) plus 1 real delete (rcv=1, old). + val v1Inserts = (1 to 10).map(i => + changeRow( + i.toLong, ('A' + i - 1).toChar.toString, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L)) + val v2Carryovers = (1 to 10).filter(_ != 5).flatMap { i => + val name = ('A' + i - 1).toChar.toString + Seq( + changeRow(i.toLong, name, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(i.toLong, name, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L)) + } + val v2RealDelete = Seq(changeRow(5L, "E", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L)) + catalog.addChangeRows(ident, v1Inserts ++ v2Carryovers ++ v2RealDelete) + + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + assert(rows.length == 1, + s"Only 1 real change should remain (9 carry-overs dropped). Got: ${descs.mkString(",")}") + assert(descs.contains("5:E:delete")) + } + + test("carry-over removal with mixed types (DOUBLE, BOOLEAN, BINARY)") { + val mixedTable = "events_mixed" + val mixedIdent = Identifier.of(Array.empty, mixedTable) + val cat = catalog + if (cat.tableExists(mixedIdent)) cat.dropTable(mixedIdent) + cat.clearChangeRows(mixedIdent) + cat.createTable( + mixedIdent, + Array( + Column.create("id", LongType), + Column.create("name", StringType), + Column.create("score", DoubleType), + Column.create("active", BooleanType), + Column.create("payload", BinaryType), + Column.create("row_commit_version", LongType, false)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + cat.setChangelogProperties(mixedIdent, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + def mixedRow( + id: Long, name: String, score: Double, active: Boolean, payload: Array[Byte], + ct: String, v: Long, rowCommitVersion: Long): InternalRow = { + InternalRow( + id, UTF8String.fromString(name), score, active, payload, rowCommitVersion, + UTF8String.fromString(ct), v, 0L) + } + + val alicePayload = Array[Byte](1, 2, 3) + val bobPayload = Array[Byte](4, 5, 6) + + cat.addChangeRows(mixedIdent, Seq( + mixedRow( + 1L, "Alice", 95.5, true, alicePayload, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + mixedRow( + 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2: update Alice's score (old rcv=1, new rcv=2); Bob is carry-over (rcv unchanged) + mixedRow( + 1L, "Alice", 95.5, true, alicePayload, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + mixedRow( + 1L, "Alice", 99.0, true, alicePayload, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L), + mixedRow( + 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + mixedRow( + 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) + + val rows = sql( + s"SELECT id, name, score, active, _change_type FROM $catalogName.$mixedTable " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => s"${r.getLong(0)}:${r.getString(4)}") + assert(descs.contains("1:update_preimage")) + assert(descs.contains("1:update_postimage")) + assert(!descs.contains("2:delete"), + s"Bob carry-over must be dropped despite DOUBLE/BOOLEAN/BINARY. Got: " + + descs.mkString(",")) + + val pre = rows.find(r => + r.getLong(0) == 1L && r.getString(4) == CHANGE_TYPE_UPDATE_PREIMAGE).get + val post = rows.find(r => + r.getLong(0) == 1L && r.getString(4) == CHANGE_TYPE_UPDATE_POSTIMAGE).get + assert(pre.getDouble(2) == 95.5) + assert(post.getDouble(2) == 99.0) + } + + // =========================================================================== + // Regression: nested rowId + nested rowVersion end-to-end + // =========================================================================== + + // End-to-end check that nested rowId paths (e.g. `payload.id`) are resolved on the plan + // and threaded through carry-over detection. The pair survives the filter because the + // row_commit_version differs across delete/insert, not because of any sibling-field data. + test("nested rowId path resolves correctly through carry-over filter") { + val nestedTable = "events_nested" + val nestedIdent = Identifier.of(Array.empty, nestedTable) + val cat = catalog + if (cat.tableExists(nestedIdent)) cat.dropTable(nestedIdent) + cat.clearChangeRows(nestedIdent) + + val payloadType = StructType(Seq( + StructField("id", LongType), + StructField("value", StringType))) + + cat.createTable( + nestedIdent, + Array( + Column.create("payload", payloadType), + Column.create("row_commit_version", LongType, false)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + + cat.setChangelogProperties(nestedIdent, ChangelogProperties( + containsCarryoverRows = true, + rowIdPaths = Seq(Seq("payload", "id")), + rowVersionName = Some("row_commit_version"))) + + def nestedRow( + id: Long, value: String, ct: String, v: Long, rowCommitVersion: Long): InternalRow = { + InternalRow( + InternalRow(id, UTF8String.fromString(value)), + rowCommitVersion, + UTF8String.fromString(ct), v, 0L) + } + + cat.addChangeRows(nestedIdent, Seq( + nestedRow(1L, "original", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2 update: rowId same, rowVersion differs (old rcv=1 on preimage, new rcv=2 on postimage) + nestedRow(1L, "original", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + nestedRow(1L, "CHANGED", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) + + val rows = sql( + s"SELECT payload.id AS id, payload.value AS value, _change_type, _commit_version " + + s"FROM $catalogName.$nestedTable CHANGES FROM VERSION 1 TO VERSION 2") + .orderBy("_commit_version", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") + + assert(descs.contains("1:original:insert:v1"), + s"v1 insert must survive. Got: ${descs.mkString(",")}") + assert(descs.contains("1:original:delete:v2"), + s"v2 delete must survive (payload.value differs from insert). Got: ${descs.mkString(",")}") + assert(descs.contains("1:CHANGED:insert:v2"), + s"v2 insert must survive (payload.value differs from delete). Got: ${descs.mkString(",")}") + assert(rows.length == 3, + s"Expected 3 rows (v1 insert + v2 delete + v2 insert). Got ${rows.length}: " + + descs.mkString(",")) + } + + // =========================================================================== + // No-op UPDATE is correctly preserved as update_preimage/postimage + // =========================================================================== + + test("no-op UPDATE is labeled as update (row_commit_version differs on pre/post)") { + // A no-op UPDATE bumps row_commit_version even when data is byte-identical, so the + // delete side carries the OLD rcv and the insert side the NEW rcv. Window post-processing + // sees different rowVersions, treats this as a real change, and labels both rows as + // update_preimage / update_postimage. + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2 no-op update: identical data, but rcv differs (Delta bumps it on any UPDATE) + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") + + assert(descs.contains("1:Alice:insert:v1")) + assert(descs.contains("1:Alice:update_preimage:v2"), + s"No-op UPDATE preimage must be labeled. Got: ${descs.mkString(",")}") + assert(descs.contains("1:Alice:update_postimage:v2"), + s"No-op UPDATE postimage must be labeled. Got: ${descs.mkString(",")}") + assert(rows.length == 3, + s"Expected v1 insert + v2 update pre/post = 3 rows. Got ${rows.length}") + } + + // =========================================================================== + // Baseline (range syntax / connector range filtering -- rule bypassed via + // deduplicationMode = 'none'; included as smoke tests for the SQL surface). + // =========================================================================== + + test("baseline: single-version range FROM VERSION X TO VERSION X") { + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (deduplicationMode = 'none')") + .collect() + + assert(rows.length == 1, s"Single version: 1 row. Got ${rows.length}") + assert(rows(0).getLong(0) == 3L) + assert(rows(0).getString(1) == CHANGE_TYPE_INSERT) + } + + test("baseline: EXCLUSIVE start bound skips the start version") { + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) + + val rows = sql( + s"SELECT id, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 EXCLUSIVE TO VERSION 3 " + + s"WITH (deduplicationMode = 'none')") + .orderBy("_commit_version") + .collect() + + assert(!rows.exists(_.getLong(1) == 1L), "v1 must be excluded") + assert(rows.exists(_.getLong(0) == 2L), "Bob (v2) included") + assert(rows.exists(_.getLong(0) == 3L), "Charlie (v3) included") + } + + test("baseline: open-ended range (no TO clause) reads to latest") { + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) + + val rows = sql( + s"SELECT id, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 WITH (deduplicationMode = 'none')") + .orderBy("_commit_version", "id") + .collect() + + assert(rows.length == 3, s"Open-ended range should see all 3. Got ${rows.length}") + assert(rows.exists(r => r.getLong(0) == 3L && r.getLong(1) == 3L)) + } +} From 697d021f3cfdb48adb4e5efbaa748bdfbc39e179 Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Tue, 28 Apr 2026 16:00:39 +0800 Subject: [PATCH 002/286] [SPARK-52729][SQL] Add MetadataOnlyTable and CREATE/ALTER VIEW support for DS v2 catalogs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? This PR exposes a DS v2 API for **metadata-only tables** (read side), **CREATE VIEW**, and **ALTER VIEW ... AS** (write side) so that third-party v2 catalogs can participate in Spark's resolution and creation flows without reimplementing read/write themselves. The headline API choice is a new **`RelationCatalog`** interface that owns the cross-cutting rules for catalogs that expose both tables and views. `TableCatalog` and `ViewCatalog` remain strict single-kind APIs; mixing the two requires `RelationCatalog`. Details in section 4. **1. Read path — `MetadataOnlyTable`:** - New `Table` implementation that carries a `TableInfo` and delegates everything to it. Catalogs return it from `TableCatalog.loadTable` for data-source tables (Spark interprets the provider via V1 paths), and from `RelationCatalog.loadRelation` wrapping a `ViewInfo` for views (Spark expands the view text). Downstream consumers distinguish via `getTableInfo() instanceof ViewInfo`. - `Analyzer.lookupTableOrView` and `RelationResolution.tryResolvePersistent` detect `MetadataOnlyTable` and route through a new `V1Table.toCatalogTable` adapter to the existing V1 data-source / view machinery. **2. Shared DTO — `TableInfo`:** - `TableInfo.Builder` gains convenience setters that write reserved keys into `properties`: `withProvider`, `withLocation`, `withComment`, `withCollation`, `withOwner`, `withTableType`, plus `withSchema(StructType)`. The read side (`MetadataOnlyTable`) and the write side (`createTable(ident, TableInfo)`) use the same struct. View-specific fields live on a typed subclass (see section 3) so they are not encoded as string properties. - `withProperties` takes a defensive copy so convenience setters don't mutate the caller's map. **3. Typed view DTO — `ViewInfo`:** - `ViewInfo extends TableInfo` carries the view-specific fields that cannot be represented as string table properties: `queryText`, `currentCatalog`, `currentNamespace` (multi-part, never `null`; empty when no namespace was captured), `sqlConfigs` (unprefixed SQL config keys), `schemaMode` (`BINDING` / `COMPENSATION` / `TYPE EVOLUTION` / `EVOLUTION`), and `queryColumnNames` (mapping query output to the view's declared columns; empty in `EVOLUTION` mode). - `ViewInfo.Builder extends TableInfo.BaseBuilder` adds typed setters: `withQueryText`, `withCurrentCatalog`, `withCurrentNamespace`, `withSqlConfigs`, `withSchemaMode`, `withQueryColumnNames`. The inherited `TableInfo.BaseBuilder` setters (schema, properties, owner, comment, collation, etc.) are available on the same builder so view and table writes share one fluent API. - The `ViewInfo` constructor stamps `PROP_TABLE_TYPE = TableSummary.VIEW_TABLE_TYPE` into `properties()` so catalogs and generic viewers reading `PROP_TABLE_TYPE` from the properties bag (e.g. `TableCatalog.listTableSummaries` default impl, `DESCRIBE`) classify the entry as `VIEW` without requiring authors to remember `withTableType(VIEW)`. - `ViewInfo` is the typed payload returned by `ViewCatalog.loadView` and accepted by `createView` / `replaceView`. It still extends `TableInfo` so a `RelationCatalog` can opt into the single-RPC perf path by returning a `MetadataOnlyTable(ViewInfo)` from `loadRelation` (see section 4); pure view-only catalogs never see `TableInfo` directly because the typed builder covers everything they construct. **4. API design — `RelationCatalog`, `ViewCatalog`, the orthogonal split:** This PR's API surface is shaped by two principles: - **Orthogonal interfaces.** Every `TableCatalog` method behaves as if views did not exist; every `ViewCatalog` method behaves as if tables did not exist. A pure `TableCatalog` impl and a pure `ViewCatalog` impl never need to know about the other kind. - **Single identifier namespace.** When a catalog exposes both, tables and views share one keyspace within a namespace; the same identifier cannot resolve to both at the same time. These two together require a third interface for the combined case — the orthogonality forbids cross-type clauses on `TableCatalog` / `ViewCatalog` methods, and the shared namespace forces cross-type rules to exist somewhere. That somewhere is `RelationCatalog`. - **`ViewCatalog`**: plugin-facing API for views. `listViews(namespace)`, `loadView(ident)` returning `ViewInfo`, `createView(ident, ViewInfo)`, `replaceView(ident, ViewInfo)`, `dropView(ident)`, default `viewExists(ident)`, default `invalidateView(ident)`, default `createOrReplaceView(ident, ViewInfo)`. The default `createOrReplaceView` impl tries `replaceView` and falls back to `createView` on `NoSuchViewException` (two RPCs, non-atomic across the pair); catalogs that can do an atomic upsert in a single transactional call should override to collapse to one RPC and make the swap atomic. No staging variant — `replaceView` is a single atomic-swap call. Class Javadoc reads as a strict view-only API and points mixed implementers at `RelationCatalog`. - **`TableCatalog`**: unchanged surface, but the `loadTable` Javadoc reverts to strictly tables-only (the perf opt-in moved off this method, see below). Cross-type clauses scattered through method docs are removed; class doc points mixed implementers at `RelationCatalog`. - **`RelationCatalog extends TableCatalog, ViewCatalog`**: the home for the combined case. Class Javadoc owns the two principles and two per-method contract tables — **active rejection** (5 write-side methods that throw on cross-type collision: `createTable`, `renameTable`, `createView`, `createOrReplaceView`, `replaceView`) and **passive filtering** (10 read / non-collision-mutation methods that behave as if the wrong kind doesn't exist). Catalogs that implement `TableCatalog` and `ViewCatalog` directly without `RelationCatalog` are rejected at `Catalogs.load` with a clear pointer at the right interface. - **Single-RPC perf entry points** on `RelationCatalog`: - `loadRelation(ident)` — the resolver's per-identifier read path. Returns a regular `Table` for a table or a `MetadataOnlyTable` wrapping a `ViewInfo` for a view; saves the cold-cache `loadTable` → `loadView` two-step. Crucially, this lives **only** on `RelationCatalog` — `TableCatalog.loadTable` is strictly tables-only, which means `tableExists`'s default impl (`loadTable(ident) != null`) is correct under any catalog with no override required. The earlier shape (perf opt-in via `loadTable` returning `MetadataOnlyTable(ViewInfo)`) silently broke `tableExists`'s default impl and was a foot-gun on three independent framework call sites. - `listRelationSummaries(namespace)` — unified listing of tables and views as `TableSummary[]` with kind preserved. Default impl unions `listTableSummaries` + `listViews` (two round trips); override to fetch in one. - **Implementer ergonomics**: `RelationCatalog` provides default impls of `loadTable`, `loadView`, `tableExists`, and `viewExists` that derive from `loadRelation` (a `MetadataOnlyTable + ViewInfo` discriminator routes the result to the right kind). A `RelationCatalog` author writes the read-side lookup once in `loadRelation`; the four kind-specific accessors come for free. They can still be overridden when a kind-specific path is materially cheaper than the unified one, but they're no longer required boilerplate. - **Resolver wiring**: `Analyzer.lookupTableOrView` and `RelationResolution.tryResolvePersistent` check `RelationCatalog` first and call `loadRelation`. Otherwise they fall through to the existing two-step (`loadTable` if `TableCatalog`, then `loadView` if `ViewCatalog`). The `MISSING_CATALOG_ABILITY.VIEWS` gate in the resolver and `CheckViewReferences` for CREATE / ALTER VIEW continues to fire for catalogs that aren't `ViewCatalog` (and therefore aren't `RelationCatalog`). **5. Write path — DS v2 CREATE VIEW:** - `DataSourceV2Strategy` routes `CreateView(ResolvedIdentifier(catalog, ident), …)` to `CreateV2ViewExec(catalog: ViewCatalog, …)`, which uses an act-then-decode pattern to keep the happy path at one RPC: plain CREATE calls `createView`; CREATE OR REPLACE calls `createOrReplaceView` (one RPC for catalogs that override the default; two for those that don't). CREATE VIEW IF NOT EXISTS short-circuits via an upfront `viewExists` probe so the view body isn't analyzed when the view already exists (matches v1 `CreateViewCommand.run`). On `ViewAlreadyExistsException` from the catalog (cross-type collision in a `RelationCatalog`, or a race for plain CREATE / OR REPLACE), the exec decodes with one `tableExists` probe: if a table is at the identifier, plain CREATE surfaces `TABLE_OR_VIEW_ALREADY_EXISTS` and CREATE OR REPLACE surfaces `EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE`, while IF NOT EXISTS is a no-op (v1-parity); if a view is at the identifier, plain CREATE re-raises as `viewAlreadyExists` and IF NOT EXISTS is a no-op (lost the race for the same view). Because `tableExists` is now contract-clean (no perf-opt-in leak), the decoder gives the right error in every catalog shape. - The exec builds the `ViewInfo` via a `V2ViewPreparation` trait reusing v1 `ViewHelper` helpers (`aliasPlan`, `sqlConfigsToProps`) to populate a `ViewInfo.Builder` with the current session's captured catalog/namespace and SQL configs. Cyclic-reference detection and auto-generated-alias rejection run once at analysis time in `CheckViewReferences` (see section 7). - `CreateView` logical plan extends `AnalysisOnlyCommand` (same shape as `V2CreateTableAsSelectPlan`) so `HandleSpecialCommand.markAsAnalyzed` captures `referredTempFunctions` from `AnalysisContext`. The v1 rewriting path (`ResolveSessionCatalog` → `CreateViewCommand`) is unchanged. **6. Write path — DS v2 ALTER VIEW ... AS:** - `AlterViewAs` logical plan also extends `AnalysisOnlyCommand` so `referredTempFunctions` is captured for the non-session path. - `DataSourceV2Strategy` routes `AlterViewAs(ResolvedPersistentView(catalog, ident, _), …)` to `AlterV2ViewExec(catalog: ViewCatalog, …)`, which calls `replaceView` (the single atomic-swap entry point — no separate staging variant, since view REPLACE writes only metadata). - A `V2AlterViewPreparation` trait (extends `V2ViewPreparation`) calls `catalog.loadView(ident)` once and uses the result to preserve user TBLPROPERTIES, comment, collation, owner, and schema-binding mode when constructing the replacement `ViewInfo`. Session-scoped fields (SQL configs, query column names) are re-emitted by `buildViewInfo()` from the active `SparkSession`, matching v1 `AlterViewAsCommand.alterPermanentView`. A racing DDL between analysis and exec (the view dropped, or replaced with a non-view table in a `RelationCatalog`) surfaces `NoSuchViewException` / `EXPECT_VIEW_NOT_TABLE` rather than a stale-resolution error. - `ResolvedViewIdentifier.unapply` (in `ResolveSessionCatalog`) replaces its `assert(isSessionCatalog)` with an `if isSessionCatalog` guard so non-session `ResolvedPersistentView` plans fall through to the v2 strategy instead of tripping the assertion. **7. Post-analysis check — `CheckViewReferences`:** - New rule wired into `BaseSessionStateBuilder.extendedCheckRules`. Rejects permanent views that reference temporary objects and rejects view bodies with auto-generated aliases for both `CreateView` and `AlterViewAs` (v2 paths). v1 `CreateViewCommand` / `AlterViewAsCommand` keep their existing exec-time safety net — Dataset-built commands can be constructed with `isAnalyzed=true` directly and bypass the analyzer's re-capture path. **8. Listing — `SHOW TABLES` / `SHOW VIEWS`:** - `TableCatalog.listTables` returns table identifiers only — views (in a `RelationCatalog`) are listed separately via `ViewCatalog.listViews`. `listTableSummaries`'s default impl enumerates via `listTables` + `loadTable` and returns one summary per table. This is an intentional v2 divergence from v1 `SHOW TABLES`, which includes both tables and views; `RelationCatalog.listRelationSummaries` is the unified entry point if a `RelationCatalog` wants to expose both in one call. Routing the `SHOW TABLES` SQL command through `listRelationSummaries` for v1-parity output is left as a follow-up so this PR's API surface stays narrowly scoped. - `SHOW VIEWS` on a non-session `ViewCatalog` is routed through a new `ShowViewsExec` that enumerates via `ViewCatalog.listViews(namespace)`. `ResolveSessionCatalog.ShowViews` skips (via guard) for `ViewCatalog` catalogs so they fall through to this strategy; non-session, non-`ViewCatalog` catalogs still hit the existing `MISSING_CATALOG_ABILITY.VIEWS` rejection. v2 catalogs have no temp views, so the `isTemporary` column is always false (mirroring v1, which only sets it true for local/global temp views). ### Why are the changes needed? A v2 `Table` is not always backed by a connector that implements read/write. Catalogs like HMS and Unity Catalog store only metadata and rely on Spark to interpret the table provider as a data source or to execute the view SQL. Previously the only way to achieve that was a hack around `V1Table`, which leaks private v1 types into v2 connectors (example: https://github.com/unitycatalog/unitycatalog/blob/main/connectors/spark/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala). Separately, v2 catalogs had no public way to handle CREATE VIEW or ALTER VIEW. `ResolveSessionCatalog` rejected CREATE VIEW on any non-session catalog with `MISSING_CATALOG_ABILITY.VIEWS`, so third-party catalogs could not own view lifecycle at all. The new `ViewCatalog` interface gives catalogs a clean view-shaped API, and the new `RelationCatalog` interface (extending both `TableCatalog` and `ViewCatalog`) is the single home for the cross-cutting rules of the combined case — orthogonality, the shared identifier namespace, the per-method active-rejection / passive-filtering tables, and the single-RPC perf entry points (`loadRelation`, `listRelationSummaries`). This separation keeps `TableCatalog` and `ViewCatalog` Javadocs as strict single-kind APIs (a connector author writing only one of them never sees cross-type clauses), and gives `RelationCatalog` implementers a single document with the full contract. ### Does this PR introduce _any_ user-facing change? Yes to connector developers: - Third-party v2 `TableCatalog` implementations can now return a `MetadataOnlyTable` from `loadTable` to delegate reads to Spark. - Third-party v2 connectors that expose **only** views implement just `ViewCatalog` (`listViews` / `loadView` / `createView` / `replaceView` / `dropView`, plus the default helpers). Connectors that expose **both** tables and views implement `RelationCatalog` (which extends both `TableCatalog` and `ViewCatalog`); implementing the two interfaces directly is rejected at catalog initialization. View text, schema, captured current catalog+namespace, SQL configs, and temp-object-reference rejection are handled the same way as for session-catalog views. No SQL-level or user-visible behavior change for existing deployments. ### Remaining work (follow-up PRs) This PR covers the core read path, CREATE VIEW (all shapes), `ALTER VIEW ... AS`, `DROP VIEW`, and `SHOW VIEWS`. The following view-scoped plans for DS v2 catalogs are **not** yet supported and are tracked for follow-ups. Until the follow-ups land, each currently surfaces a clean `UNSUPPORTED_FEATURE.TABLE_OPERATION` error (wired up in `DataSourceV2Strategy` and pinned by tests in `DataSourceV2MetadataOnlyViewSuite`), so users get a meaningful message rather than a generic planner failure: - **`ALTER VIEW ... SET/UNSET TBLPROPERTIES`** — separate logical plans (`SetViewProperties`, `UnsetViewProperties`); need their own `DataSourceV2Strategy` cases backed by new `TableChange` routing. - **`ALTER VIEW ... RENAME TO`** — `RenameTable` at the logical level; needs v2 view awareness and the catalog-side rename semantics. - **`ALTER VIEW ... WITH SCHEMA BINDING`** — `AlterViewSchemaBinding` logical plan; needs the same treatment as `ALTER VIEW AS` (AnalysisOnlyCommand shape + v2 exec). - **`DESCRIBE` / `SHOW CREATE TABLE` / `SHOW TBLPROPERTIES` / `SHOW COLUMNS`** on v2 views — currently route through `ResolvedViewIdentifier` which only matches session-catalog views; the v2 equivalents need dedicated handling. - **`SHOW TABLES`** v1-parity output (include views) on a v2 catalog — `TableCatalog.listTables` now intentionally returns tables only; routing the SQL `SHOW TABLES` through `RelationCatalog.listRelationSummaries` for v1-parity is a separate piece of work. ### How was this patch tested? Two new test suites: `DataSourceV2MetadataOnlyTableSuite` (4 tests, table-side) and `DataSourceV2MetadataOnlyViewSuite` (56 tests, view-side + mixed-catalog). #### `DataSourceV2MetadataOnlyTableSuite` — table side - **File-source reads:** non-partitioned and partitioned `MetadataOnlyTable` round-trip through SELECT / INSERT / INSERT OVERWRITE. - **`DESCRIBE TABLE EXTENDED`:** `Name` row shows the real catalog-qualified identifier, not the wrapper class. - **Fully-qualified column references:** 3-part `catalog.schema.table.col` references resolve correctly through a `MetadataOnlyTable`. The mechanism: `V1Table.toCatalogTable` sets `CatalogTable.multipartIdentifier = [catalog, namespace…, table]`, and the `SessionCatalog` change in this PR makes `getRelation` prefer that over the hardcoded `spark_catalog` qualifier when wrapping in a `SubqueryAlias`. (1-part and 2-part references already worked via last-part suffix matching; the test pins both cases plus the 3-part case to make the alias-source change visible in the diff.) #### `DataSourceV2MetadataOnlyViewSuite` — view side + mixed catalogs Catalog fixtures: `TestingRelationCatalog` (`RelationCatalog`), `TestingViewOnlyCatalog` (pure `ViewCatalog`), `TestingTableOnlyCatalog` (pure `TableCatalog`). - **Read path** — view-text expansion with captured SQL configs; unqualified references resolved via single-part and multi-part captured `currentCatalog` + `currentNamespace`; pure `ViewCatalog` read path (no `TableCatalog` mixin) confirms the resolver's `loadView` fallback fires when `loadTable` is skipped. - **`V1Table.toCatalogTable(ViewInfo)` round-trip** — multi-part captured namespace flows through structurally (including parts containing dots, which would break a string-encoded path); absent-`currentCatalog` branch yields an empty `viewCatalogAndNamespace`. - **CREATE VIEW** — end-to-end CREATE; `CREATE VIEW IF NOT EXISTS` no-op vs. failure; `CREATE OR REPLACE VIEW` replacement (including cyclic-reference detection); too-few / too-many user-specified columns; rejection of references to temp function / temp view / temp variable; `DEFAULT COLLATION` propagation into `ViewInfo`; cross-type collision over a non-view table entry surfaces `TABLE_OR_VIEW_ALREADY_EXISTS` (plain) / `EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE` (OR REPLACE) / no-op (IF NOT EXISTS) — v1-parity; `PROP_OWNER` stamped on stored `TableInfo`; rejection on a `TableCatalog`-only catalog with `MISSING_CATALOG_ABILITY.VIEWS`. - **ALTER VIEW … AS** — body replacement; missing-view fails at analysis; rejection of references to temp function / temp view / temp variable; preserves user-set TBLPROPERTIES, `PROP_OWNER`, and `SCHEMA EVOLUTION` binding mode (v1-parity); re-captures the current session's SQL configs; cyclic-reference detection; rejection on a `TableCatalog`-only catalog with `MISSING_CATALOG_ABILITY.VIEWS`; pure `ViewCatalog` flow; `CREATE OR REPLACE VIEW` whose new body references a nonexistent table fails at the right phase. - **Multi-level namespace handling** — cyclic-reference detection distinguishes views across multi-level namespaces (CREATE and ALTER); error messages render the full multi-level namespace; temp-object reference errors render the full multi-level namespace. - **DROP VIEW** — drops a view on `ViewCatalog`; `DROP VIEW IF EXISTS` is a no-op on missing; rejection on a non-view table entry (v1-parity, `EXPECT_VIEW_NOT_TABLE`); rejection on a non-`ViewCatalog` catalog. - **SHOW TABLES / SHOW VIEWS** — `SHOW TABLES` on a v2 catalog returns tables only; `SHOW VIEWS` returns views only (`isTemporary=false` throughout for v2); `SHOW VIEWS … LIKE` filters by name; rejection on a non-`ViewCatalog` catalog. - **Unsupported v2 view DDL / inspection pinning** — `ALTER VIEW SET/UNSET TBLPROPERTIES`, `ALTER VIEW WITH SCHEMA`, `ALTER VIEW RENAME TO`, `SHOW CREATE TABLE`, `SHOW TBLPROPERTIES`, `SHOW COLUMNS`, `DESCRIBE TABLE`, `REFRESH TABLE`, `ANALYZE TABLE`, `ANALYZE TABLE … FOR COLUMNS` against a v2 view all surface `UNSUPPORTED_FEATURE.TABLE_OPERATION`; `DESCRIBE TABLE … COLUMN` surfaces a clean `AnalysisException`. Pins the current failure mode so a future regression to a generic planner error is caught in the diff. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (Anthropic) Closes #51419 from cloud-fan/v1-v2. Authored-by: Wenchen Fan Signed-off-by: Wenchen Fan --- .../connector/catalog/MetadataOnlyTable.java | 97 ++ .../connector/catalog/RelationCatalog.java | 231 ++++ .../sql/connector/catalog/TableCatalog.java | 74 +- .../sql/connector/catalog/TableInfo.java | 98 +- .../spark/sql/connector/catalog/View.java | 74 -- .../sql/connector/catalog/ViewCatalog.java | 205 ++- .../sql/connector/catalog/ViewChange.java | 79 -- .../spark/sql/connector/catalog/ViewInfo.java | 227 ++-- .../sql/catalyst/analysis/Analyzer.scala | 80 +- .../analysis/ApplyDefaultCollation.scala | 4 +- .../analysis/RelationResolution.scala | 105 +- .../catalyst/analysis/ViewResolution.scala | 2 +- .../analysis/resolver/ViewResolver.scala | 2 +- .../sql/catalyst/catalog/SessionCatalog.scala | 13 +- .../sql/catalyst/catalog/interface.scala | 48 +- .../catalyst/plans/logical/v2Commands.scala | 77 +- .../catalog/CatalogV2Implicits.scala | 9 + .../sql/connector/catalog/Catalogs.scala | 19 + .../spark/sql/connector/catalog/V1Table.scala | 107 +- .../sql/errors/QueryCompilationErrors.scala | 50 +- .../analysis/ResolveSessionCatalog.scala | 43 +- .../command/metricViewCommands.scala | 7 +- .../spark/sql/execution/command/views.scala | 137 +- .../datasources/v2/AlterV2ViewExec.scala | 115 ++ .../datasources/v2/CreateV2ViewExec.scala | 175 +++ .../datasources/v2/DataSourceV2Strategy.scala | 96 +- .../datasources/v2/DropTableExec.scala | 27 +- .../datasources/v2/DropViewExec.scala | 60 + .../datasources/v2/ShowViewsExec.scala | 51 + .../internal/BaseSessionStateBuilder.scala | 3 +- .../analyzer-results/explain-aqe.sql.out | 2 +- .../analyzer-results/explain.sql.out | 2 +- .../DataSourceV2MetadataOnlyTableSuite.scala | 181 +++ .../DataSourceV2MetadataOnlyViewSuite.scala | 1120 +++++++++++++++++ .../sql/connector/DataSourceV2SQLSuite.scala | 6 +- .../command/PlanResolutionSuite.scala | 4 +- .../execution/command/v2/DropTableSuite.scala | 7 + 37 files changed, 3009 insertions(+), 628 deletions(-) create mode 100644 sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataOnlyTable.java create mode 100644 sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/RelationCatalog.java delete mode 100644 sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/View.java delete mode 100644 sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewChange.java create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterV2ViewExec.scala create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateV2ViewExec.scala create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropViewExec.scala create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowViewsExec.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataOnlyTableSuite.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataOnlyViewSuite.scala diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataOnlyTable.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataOnlyTable.java new file mode 100644 index 0000000000000..b20a9b566646f --- /dev/null +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataOnlyTable.java @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector.catalog; + +import java.util.Collections; +import java.util.Map; +import java.util.Objects; +import java.util.Set; + +import org.apache.spark.annotation.Evolving; +import org.apache.spark.sql.connector.catalog.constraints.Constraint; +import org.apache.spark.sql.connector.expressions.Transform; + +/** + * A concrete {@code Table} implementation that contains only table metadata, deferring + * read/write to Spark. It represents a general Spark data source table or a Spark view; + * Spark resolves the table provider into a data source (for tables) or expands the view text + * (for views) at read time. + *

+ * Catalogs build the metadata via {@link TableInfo.Builder} (for data-source tables) or + * {@link ViewInfo.Builder} (for views). A {@code MetadataOnlyTable} wrapping a + * {@link TableInfo} can be returned from {@link TableCatalog#loadTable(Identifier)} for a + * data-source table; a {@code MetadataOnlyTable} wrapping a {@link ViewInfo} can be returned + * from {@link RelationCatalog#loadRelation(Identifier)} as the single-RPC perf opt-in for a view. + * Downstream consumers distinguish the two by checking + * {@code getTableInfo() instanceof ViewInfo}. + * + * @since 4.2.0 + */ +@Evolving +public class MetadataOnlyTable implements Table { + private final TableInfo info; + private final String name; + + /** + * @param info metadata for the table or view. Pass a {@link ViewInfo} for a view. + * @param name human-readable name for this table, used by places that read {@link #name()} + * (e.g. the {@code Name} row of {@code DESCRIBE TABLE EXTENDED}). Catalogs + * returning a {@code MetadataOnlyTable} from {@link TableCatalog#loadTable} or + * {@link RelationCatalog#loadRelation} should typically pass + * {@code ident.toString()}, matching the quoted multi-part form used elsewhere + * for v2 identifiers. + */ + public MetadataOnlyTable(TableInfo info, String name) { + this.info = Objects.requireNonNull(info, "info should not be null"); + this.name = Objects.requireNonNull(name, "name should not be null"); + } + + public TableInfo getTableInfo() { + return info; + } + + @Override + public Column[] columns() { + return info.columns(); + } + + @Override + public Map properties() { + return Collections.unmodifiableMap(info.properties()); + } + + @Override + public Transform[] partitioning() { + return info.partitions(); + } + + @Override + public Constraint[] constraints() { + return info.constraints(); + } + + @Override + public String name() { + return name; + } + + @Override + public Set capabilities() { + return Set.of(); + } +} diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/RelationCatalog.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/RelationCatalog.java new file mode 100644 index 0000000000000..bb674faa10ac5 --- /dev/null +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/RelationCatalog.java @@ -0,0 +1,231 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.connector.catalog; + +import java.util.ArrayList; + +import org.apache.spark.annotation.Evolving; +import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException; +import org.apache.spark.sql.catalyst.analysis.NoSuchTableException; +import org.apache.spark.sql.catalyst.analysis.NoSuchViewException; + +/** + * Catalog API for connectors that expose both tables and views in a single shared identifier + * namespace. + *

+ * Connectors that expose both tables and views must implement {@code RelationCatalog}; + * implementing {@link TableCatalog} and {@link ViewCatalog} directly without + * {@code RelationCatalog} is rejected at catalog initialization. Connectors that expose only + * tables implement just {@link TableCatalog}; connectors that expose only views implement just + * {@link ViewCatalog}; this interface is not relevant to them. + * + *

Two principles

+ * + * A {@code RelationCatalog} follows two rules that, taken together, define every cross-cutting + * subtlety: + *
    + *
  1. Orthogonal interfaces. Every {@link TableCatalog} method behaves as if views did + * not exist, and every {@link ViewCatalog} method behaves as if tables did not exist. + * From the perspective of a {@code TableCatalog} caller, a view at an identifier is + * indistinguishable from "nothing there"; symmetrically for {@code ViewCatalog} on + * tables. The implementation, of course, knows about both kinds -- it just filters them + * apart at each method boundary.
    2. + *
  2. Single identifier namespace. Tables and views share one keyspace within a + * namespace; the same {@link Identifier} cannot resolve to both at the same time. The + * implementation typically enforces this with a single backing keyspace plus a kind + * discriminator.
    3. + *
+ * + *

Per-method cross-type behavior

+ * + * Active rejection (write-side methods that throw on cross-type collision): + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
Cross-type rejection
MethodRejects whenThrows
{@link TableCatalog#createTable}a view sits at {@code ident}{@link org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException}
{@link TableCatalog#renameTable}a view sits at {@code newIdent}{@link org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException}
{@link ViewCatalog#createView}a table sits at {@code ident}{@link org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException}
{@link ViewCatalog#createOrReplaceView}a table sits at {@code ident}{@link org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException}
{@link ViewCatalog#replaceView}a table sits at {@code ident}{@link org.apache.spark.sql.catalyst.analysis.NoSuchViewException}
+ * + * Passive filtering (read / non-collision mutation methods that behave as if the wrong + * kind doesn't exist): + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
Cross-type filtering
MethodOn wrong-kind ident
{@link TableCatalog#loadTable(Identifier)}throws {@code NoSuchTableException} for a view
{@link TableCatalog#loadTable(Identifier, String)} / + * {@link TableCatalog#loadTable(Identifier, long)}throws {@code NoSuchTableException} for a view (no perf opt-in -- time-travel does + * not apply to views)
{@link TableCatalog#tableExists}returns {@code false} for a view
{@link TableCatalog#dropTable} / {@link TableCatalog#purgeTable}returns {@code false} for a view; does not drop it
{@link TableCatalog#renameTable}throws {@code NoSuchTableException} when the source is a view
{@link TableCatalog#listTables}tables only
{@link ViewCatalog#loadView}throws {@code NoSuchViewException} for a table
{@link ViewCatalog#viewExists}returns {@code false} for a table
{@link ViewCatalog#dropView}returns {@code false} for a table; does not drop it
{@link ViewCatalog#listViews}views only
+ * + *

Single-RPC perf entry points

+ * + * The orthogonal {@link TableCatalog} and {@link ViewCatalog} answer two cross-cutting + * questions in two round trips each. {@code RelationCatalog} adds dedicated methods so a + * catalog can answer both in one round trip: + *
    + *
  • {@link #loadRelation(Identifier)} -- the resolver's per-identifier read path. Returns + * a regular {@link Table} for a table, or a {@link MetadataOnlyTable} wrapping a + * {@link ViewInfo} for a view. Saves the {@code loadTable} -> {@code loadView} fallback + * on a cold cache.
    • + *
  • {@link #listRelationSummaries(String[])} -- a unified listing of tables and views with the + * kind preserved on each {@link TableSummary}. Default impl performs both + * {@link TableCatalog#listTableSummaries} and {@link ViewCatalog#listViews}; override to + * fetch in one round trip.
    • + *
+ * + * @since 4.2.0 + */ +@Evolving +public interface RelationCatalog extends TableCatalog, ViewCatalog { + + /** + * Load metadata for an identifier that may resolve to either a table or a view. + *

+ * For a table, returns the table's {@link Table}. For a view, returns a + * {@link MetadataOnlyTable} wrapping a {@link ViewInfo}; callers discriminate via + * {@code getTableInfo() instanceof ViewInfo}. This lets the resolver answer in a single RPC + * instead of falling back from {@link TableCatalog#loadTable} to {@link ViewCatalog#loadView}. + * + * @param ident the identifier + * @return a {@link Table} for tables, or a {@link MetadataOnlyTable} wrapping a + * {@link ViewInfo} for views + * @throws NoSuchTableException if neither a table nor a view exists at {@code ident} + */ + Table loadRelation(Identifier ident) throws NoSuchTableException; + + /** + * List the tables and views in a namespace, returned as {@link TableSummary} entries with + * the kind preserved on each summary. + *

+ * The default implementation enumerates via {@link TableCatalog#listTableSummaries} for + * tables and {@link ViewCatalog#listViews} for views (two round trips). Catalogs that can + * fetch the unified listing in a single round trip should override. + * + * @param namespace a multi-part namespace + * @return an array of summaries for both tables and views in the namespace + * @throws NoSuchNamespaceException if the namespace does not exist (optional) + * @throws NoSuchTableException if a table listed by the underlying enumeration disappears + * before its summary can be assembled (default impl only) + */ + default TableSummary[] listRelationSummaries(String[] namespace) + throws NoSuchNamespaceException, NoSuchTableException { + TableSummary[] tableSummaries = listTableSummaries(namespace); + Identifier[] viewIdentifiers = listViews(namespace); + ArrayList all = new ArrayList<>( + tableSummaries.length + viewIdentifiers.length); + for (TableSummary s : tableSummaries) { + all.add(s); + } + for (Identifier id : viewIdentifiers) { + all.add(TableSummary.of(id, TableSummary.VIEW_TABLE_TYPE)); + } + return all.toArray(TableSummary[]::new); + } + + /** + * {@inheritDoc} + *

+ * The default implementation derives from {@link #loadRelation}: a {@link MetadataOnlyTable} + * wrapping a {@link ViewInfo} is rejected as not-a-table; anything else is returned. Override + * only if a tables-only path is materially cheaper than the unified one. + */ + @Override + default Table loadTable(Identifier ident) throws NoSuchTableException { + Table t = loadRelation(ident); + if (t instanceof MetadataOnlyTable mot && mot.getTableInfo() instanceof ViewInfo) { + throw new NoSuchTableException(ident); + } + return t; + } + + /** + * {@inheritDoc} + *

+ * The default implementation derives from {@link #loadRelation}: a {@link MetadataOnlyTable} + * wrapping a {@link ViewInfo} is unwrapped and returned; anything else (table or absent) is + * surfaced as {@link NoSuchViewException}. Override only if a views-only path is materially + * cheaper than the unified one. + */ + @Override + default ViewInfo loadView(Identifier ident) throws NoSuchViewException { + Table t; + try { + t = loadRelation(ident); + } catch (NoSuchTableException e) { + throw new NoSuchViewException(ident); + } + if (t instanceof MetadataOnlyTable mot && mot.getTableInfo() instanceof ViewInfo vi) { + return vi; + } + throw new NoSuchViewException(ident); + } + + /** + * {@inheritDoc} + *

+ * The default implementation derives from {@link #loadRelation}: returns {@code true} only if + * the entry exists and is not a view. Override only if a cheaper existence-check path exists. + */ + @Override + default boolean tableExists(Identifier ident) { + try { + Table t = loadRelation(ident); + return !(t instanceof MetadataOnlyTable mot && mot.getTableInfo() instanceof ViewInfo); + } catch (NoSuchTableException e) { + return false; + } + } + + /** + * {@inheritDoc} + *

+ * The default implementation derives from {@link #loadRelation}: returns {@code true} only if + * the entry exists and is a view. Override only if a cheaper existence-check path exists. + */ + @Override + default boolean viewExists(Identifier ident) { + try { + Table t = loadRelation(ident); + return t instanceof MetadataOnlyTable mot && mot.getTableInfo() instanceof ViewInfo; + } catch (NoSuchTableException e) { + return false; + } + } +} diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java index d5a36cd8bfb86..55894357f19d1 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java @@ -31,13 +31,18 @@ import java.util.Set; /** - * Catalog methods for working with Tables. + * Catalog API for connectors that expose tables. + *

+ * Connectors that expose only tables implement this interface. Connectors that expose + * both tables and views must implement {@link RelationCatalog} (which extends both this + * interface and {@link ViewCatalog} and adds the cross-cutting contract for the combined + * case); the methods on this interface remain table-only -- they do not interact with views. *

* TableCatalog implementations may be case-sensitive or case-insensitive. Spark will pass * {@link Identifier table identifiers} without modification. Field names passed to - * {@link #alterTable(Identifier, TableChange...)} will be normalized to match the case used in the - * table schema when updating, renaming, or dropping existing columns when catalyst analysis is - * case-insensitive. + * {@link #alterTable(Identifier, TableChange...)} will be normalized to match the case used in + * the table schema when updating, renaming, or dropping existing columns when catalyst + * analysis is case-insensitive. * * @since 3.0.0 */ @@ -99,8 +104,6 @@ public interface TableCatalog extends CatalogPlugin { /** * List the tables in a namespace from the catalog. - *

- * If the catalog supports views, this must return identifiers for only tables and not views. * * @param namespace a multi-part namespace * @return an array of Identifiers for tables @@ -111,11 +114,14 @@ public interface TableCatalog extends CatalogPlugin { /** * List the table summaries in a namespace from the catalog. *

- * This method should return all tables entities from a catalog regardless of type (i.e. views - * should be listed as well). + * Returns one summary per entry returned by {@link #listTables}. Each {@link TableSummary} + * carries the entry's {@code tableType}. + *

+ * The default implementation enumerates via {@link #listTables} + {@link #loadTable}. + * Catalogs that can fetch summaries in a single round-trip should override. * * @param namespace a multi-part namespace - * @return an array of Identifiers for tables + * @return an array of summaries for tables in the namespace * @throws NoSuchNamespaceException If the namespace does not exist (optional). * @throws NoSuchTableException If certain table listed by listTables API does not exist. */ @@ -139,27 +145,21 @@ default TableSummary[] listTableSummaries(String[] namespace) /** * Load table metadata by {@link Identifier identifier} from the catalog. - *

- * If the catalog supports views and contains a view for the identifier and not a table, this - * must throw {@link NoSuchTableException}. * * @param ident a table identifier * @return the table's metadata - * @throws NoSuchTableException If the table doesn't exist or is a view + * @throws NoSuchTableException If the table doesn't exist */ Table loadTable(Identifier ident) throws NoSuchTableException; /** * Load table metadata by {@link Identifier identifier} from the catalog. Spark will write data * into this table later. - *

- * If the catalog supports views and contains a view for the identifier and not a table, this - * must throw {@link NoSuchTableException}. * * @param ident a table identifier * @param writePrivileges * @return the table's metadata - * @throws NoSuchTableException If the table doesn't exist or is a view + * @throws NoSuchTableException If the table doesn't exist * * @since 3.5.3 */ @@ -171,14 +171,11 @@ default Table loadTable( /** * Load table metadata of a specific version by {@link Identifier identifier} from the catalog. - *

- * If the catalog supports views and contains a view for the identifier and not a table, this - * must throw {@link NoSuchTableException}. * * @param ident a table identifier * @param version version of the table * @return the table's metadata - * @throws NoSuchTableException If the table doesn't exist or is a view + * @throws NoSuchTableException If the table doesn't exist */ default Table loadTable(Identifier ident, String version) throws NoSuchTableException { throw QueryCompilationErrors.noSuchTableError(name(), ident); @@ -186,14 +183,11 @@ default Table loadTable(Identifier ident, String version) throws NoSuchTableExce /** * Load table metadata at a specific time by {@link Identifier identifier} from the catalog. - *

- * If the catalog supports views and contains a view for the identifier and not a table, this - * must throw {@link NoSuchTableException}. * * @param ident a table identifier * @param timestamp timestamp of the table, which is microseconds since 1970-01-01 00:00:00 UTC * @return the table's metadata - * @throws NoSuchTableException If the table doesn't exist or is a view + * @throws NoSuchTableException If the table doesn't exist */ default Table loadTable(Identifier ident, long timestamp) throws NoSuchTableException { throw QueryCompilationErrors.noSuchTableError(name(), ident); @@ -232,12 +226,9 @@ default void invalidateTable(Identifier ident) { /** * Test whether a table exists using an {@link Identifier identifier} from the catalog. - *

- * If the catalog supports views and contains a view for the identifier and not a table, this - * must return false. * * @param ident a table identifier - * @return true if the table exists, false otherwise + * @return true if a table exists at {@code ident}, false otherwise */ default boolean tableExists(Identifier ident) { try { @@ -281,11 +272,11 @@ default Table createTable( * Create a table in the catalog. * * @param ident a table identifier - * @param tableInfo information about the table. + * @param tableInfo information about the table * @return metadata for the new table. This can be null if getting the metadata for the new table * is expensive. Spark will call {@link #loadTable(Identifier)} if needed (e.g. CTAS). * - * @throws TableAlreadyExistsException If a table or view already exists for the identifier + * @throws TableAlreadyExistsException If a table already exists for the identifier * @throws UnsupportedOperationException If a requested partition transform is not supported * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional) * @since 4.1.0 @@ -317,7 +308,7 @@ default Table createTable(Identifier ident, TableInfo tableInfo) * or other custom state from this object to clone additional metadata * @return metadata for the new table * - * @throws TableAlreadyExistsException If a table or view already exists for the identifier + * @throws TableAlreadyExistsException If a table already exists for the identifier * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional) * @throws UnsupportedOperationException If the catalog does not support CREATE TABLE LIKE * @since 4.2.0 @@ -343,16 +334,13 @@ default boolean useNullableQuerySchema() { * changes should be applied to the table. *

* The requested changes must be applied in the order given. - *

- * If the catalog supports views and contains a view for the identifier and not a table, this - * must throw {@link NoSuchTableException}. * * @param ident a table identifier * @param changes changes to apply to the table * @return updated metadata for the table. This can be null if getting the metadata for the * updated table is expensive. Spark always discard the returned table here. * - * @throws NoSuchTableException If the table doesn't exist or is a view + * @throws NoSuchTableException If the table doesn't exist * @throws IllegalArgumentException If any change is rejected by the implementation. */ Table alterTable( @@ -361,9 +349,6 @@ Table alterTable( /** * Drop a table in the catalog. - *

- * If the catalog supports views and contains a view for the identifier and not a table, this - * must not drop the view and must return false. * * @param ident a table identifier * @return true if a table was deleted, false if no table exists for the identifier @@ -374,9 +359,6 @@ Table alterTable( * Drop a table in the catalog and completely remove its data by skipping a trash even if it is * supported. *

- * If the catalog supports views and contains a view for the identifier and not a table, this - * must not drop the view and must return false. - *

* If the catalog supports to purge a table, this method should be overridden. * The default implementation throws {@link UnsupportedOperationException}. * @@ -393,17 +375,13 @@ default boolean purgeTable(Identifier ident) throws UnsupportedOperationExceptio /** * Renames a table in the catalog. *

- * If the catalog supports views and contains a view for the old identifier and not a table, this - * throws {@link NoSuchTableException}. Additionally, if the new identifier is a table or a view, - * this throws {@link TableAlreadyExistsException}. - *

* If the catalog does not support table renames between namespaces, it throws * {@link UnsupportedOperationException}. * * @param oldIdent the table identifier of the existing table to rename * @param newIdent the new table identifier of the table - * @throws NoSuchTableException If the table to rename doesn't exist or is a view - * @throws TableAlreadyExistsException If the new table name already exists or is a view + * @throws NoSuchTableException If the table to rename doesn't exist + * @throws TableAlreadyExistsException If the new table name already exists * @throws UnsupportedOperationException If the namespaces of old and new identifiers do not * match (optional) */ diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java index 9870a3b0fa45d..89709c9f1c2f0 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java @@ -33,9 +33,8 @@ public class TableInfo { /** * Constructor for TableInfo used by the builder. - * @param builder Builder. */ - private TableInfo(Builder builder) { + protected TableInfo(BaseBuilder builder) { this.columns = builder.columns; this.properties = builder.properties; this.partitions = builder.partitions; @@ -60,35 +59,96 @@ public Transform[] partitions() { public Constraint[] constraints() { return constraints; } - public static class Builder { - private Column[] columns = new Column[0]; - private Map properties = new HashMap<>(); - private Transform[] partitions = new Transform[0]; - private Constraint[] constraints = new Constraint[0]; + public static class Builder extends BaseBuilder { + @Override + protected Builder self() { return this; } - public Builder withColumns(Column[] columns) { + @Override + public TableInfo build() { + Objects.requireNonNull(columns, "columns should not be null"); + return new TableInfo(this); + } + } + + /** + * Shared builder state for {@link TableInfo} and its subclasses. Setters return {@code B} so + * subclass builders (e.g. {@link ViewInfo.Builder}) chain through their own type without + * a covariant override on each inherited setter. + */ + protected abstract static class BaseBuilder> { + protected Column[] columns = new Column[0]; + protected Map properties = new HashMap<>(); + protected Transform[] partitions = new Transform[0]; + protected Constraint[] constraints = new Constraint[0]; + + protected abstract B self(); + + public B withColumns(Column[] columns) { this.columns = columns; - return this; + return self(); } - public Builder withProperties(Map properties) { - this.properties = properties; - return this; + public B withSchema(StructType schema) { + this.columns = CatalogV2Util.structTypeToV2Columns(schema); + return self(); } - public Builder withPartitions(Transform[] partitions) { + /** + * Replaces the current properties map with a defensive copy of the given map. Any reserved + * keys set earlier via convenience setters (e.g. {@link #withProvider}) are discarded -- + * call those setters after this method, not before. + */ + public B withProperties(Map properties) { + this.properties = new HashMap<>(properties); + return self(); + } + + public B withPartitions(Transform[] partitions) { this.partitions = partitions; - return this; + return self(); } - public Builder withConstraints(Constraint[] constraints) { + public B withConstraints(Constraint[] constraints) { this.constraints = constraints; - return this; + return self(); } - public TableInfo build() { - Objects.requireNonNull(columns, "columns should not be null"); - return new TableInfo(this); + // Convenience setters below write reserved keys into the current `properties` map. Pair + // each with a preceding `withProperties(...)` call if you want to start from a user map; + // calling `withProperties` after a convenience setter discards the value the convenience + // setter wrote. + + /** Writes {@link TableCatalog#PROP_PROVIDER} into the current properties map. */ + public B withProvider(String provider) { + properties.put(TableCatalog.PROP_PROVIDER, provider); + return self(); + } + + public B withLocation(String location) { + properties.put(TableCatalog.PROP_LOCATION, location); + return self(); } + + public B withComment(String comment) { + properties.put(TableCatalog.PROP_COMMENT, comment); + return self(); + } + + public B withCollation(String collation) { + properties.put(TableCatalog.PROP_COLLATION, collation); + return self(); + } + + public B withOwner(String owner) { + properties.put(TableCatalog.PROP_OWNER, owner); + return self(); + } + + public B withTableType(String tableType) { + properties.put(TableCatalog.PROP_TABLE_TYPE, tableType); + return self(); + } + + public abstract TableInfo build(); } } diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/View.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/View.java deleted file mode 100644 index a4dc5f2f2d20f..0000000000000 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/View.java +++ /dev/null @@ -1,74 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.spark.sql.connector.catalog; - -import java.util.Map; - -import org.apache.spark.annotation.DeveloperApi; -import org.apache.spark.sql.types.StructType; - -/** - * An interface representing a persisted view. - */ -@DeveloperApi -public interface View { - /** - * A name to identify this view. - */ - String name(); - - /** - * The view query SQL text. - */ - String query(); - - /** - * The current catalog when the view is created. - */ - String currentCatalog(); - - /** - * The current namespace when the view is created. - */ - String[] currentNamespace(); - - /** - * The schema for the view when the view is created after applying column aliases. - */ - StructType schema(); - - /** - * The output column names of the query that creates this view. - */ - String[] queryColumnNames(); - - /** - * The view column aliases. - */ - String[] columnAliases(); - - /** - * The view column comments. - */ - String[] columnComments(); - - /** - * The view properties. - */ - Map properties(); -} diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewCatalog.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewCatalog.java index abe5fb3148d08..184676023d7c4 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewCatalog.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewCatalog.java @@ -14,186 +14,135 @@ * See the License for the specific language governing permissions and * limitations under the License. */ - package org.apache.spark.sql.connector.catalog; -import java.util.Arrays; -import java.util.List; - -import org.apache.spark.annotation.DeveloperApi; +import org.apache.spark.annotation.Evolving; import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException; import org.apache.spark.sql.catalyst.analysis.NoSuchViewException; import org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException; /** - * Catalog methods for working with views. + * Catalog API for connectors that expose views. + *

+ * Connectors that expose only views implement this interface. Connectors that expose + * both tables and views must implement {@link RelationCatalog} (which extends both this + * interface and {@link TableCatalog} and adds the cross-cutting contract for the combined + * case); the methods on this interface remain view-only -- they do not interact with tables. + *

+ * The presence of {@code ViewCatalog} on the catalog plugin is the signal that it + * supports views; there is no capability flag to declare. + * + * @since 4.2.0 */ -@DeveloperApi +@Evolving public interface ViewCatalog extends CatalogPlugin { - /** - * A reserved property to specify the description of the view. - */ - String PROP_COMMENT = "comment"; - - /** - * A reserved property to specify the owner of the view. - */ - String PROP_OWNER = "owner"; - - /** - * A reserved property to specify the software version used to create the view. - */ - String PROP_CREATE_ENGINE_VERSION = "create_engine_version"; - - /** - * A reserved property to specify the software version used to change the view. - */ - String PROP_ENGINE_VERSION = "engine_version"; - - /** - * All reserved properties of the view. - */ - List RESERVED_PROPERTIES = Arrays.asList( - PROP_COMMENT, - PROP_OWNER, - PROP_CREATE_ENGINE_VERSION, - PROP_ENGINE_VERSION); - /** * List the views in a namespace from the catalog. - *

- * If the catalog supports tables, this must return identifiers for only views and not tables. * * @param namespace a multi-part namespace - * @return an array of Identifiers for views - * @throws NoSuchNamespaceException If the namespace does not exist (optional). + * @return an array of identifiers for views + * @throws NoSuchNamespaceException if the namespace does not exist (optional) */ - Identifier[] listViews(String... namespace) throws NoSuchNamespaceException; + Identifier[] listViews(String[] namespace) throws NoSuchNamespaceException; /** - * Load view metadata by {@link Identifier ident} from the catalog. - *

- * If the catalog supports tables and contains a table for the identifier and not a view, - * this must throw {@link NoSuchViewException}. + * Load view metadata by identifier. * * @param ident a view identifier - * @return the view description - * @throws NoSuchViewException If the view doesn't exist or is a table + * @return the view metadata + * @throws NoSuchViewException if the view does not exist */ - View loadView(Identifier ident) throws NoSuchViewException; + ViewInfo loadView(Identifier ident) throws NoSuchViewException; /** - * Invalidate cached view metadata for an {@link Identifier identifier}. + * Test whether a view exists. *

- * If the view is already loaded or cached, drop cached data. If the view does not exist or is - * not cached, do nothing. Calling this method should not query remote services. + * The default implementation calls {@link #loadView} and catches {@link NoSuchViewException}. + * Catalogs that can answer existence cheaply should override. * * @param ident a view identifier - */ - default void invalidateView(Identifier ident) { - } - - /** - * Test whether a view exists using an {@link Identifier identifier} from the catalog. - *

- * If the catalog supports views and contains a view for the identifier and not a table, - * this must return false. - * - * @param ident a view identifier - * @return true if the view exists, false otherwise + * @return true if a view exists at {@code ident}, false otherwise */ default boolean viewExists(Identifier ident) { try { - return loadView(ident) != null; + loadView(ident); + return true; } catch (NoSuchViewException e) { return false; } } /** - * Create a view in the catalog. + * Invalidate cached metadata for a view. + *

+ * If the view is currently cached, drop the cached entry; otherwise do nothing. This must not + * issue remote calls. * - * @param viewInfo the info class holding all view information - * @return the created view. This can be null if getting the metadata for the view is expensive - * @throws ViewAlreadyExistsException If a view or table already exists for the identifier - * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional) + * @param ident a view identifier */ - View createView(ViewInfo viewInfo) throws ViewAlreadyExistsException, NoSuchNamespaceException; + default void invalidateView(Identifier ident) { + } /** - * Replace a view in the catalog. - *

- * The default implementation has a race condition. - * Catalogs are encouraged to implement this operation atomically. + * Create a view. * - * @param viewInfo the info class holding all view information - * @param orCreate create the view if it doesn't exist - * @return the created/replaced view. This can be null if getting the metadata - * for the view is expensive - * @throws NoSuchViewException If the view doesn't exist or is a table - * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional) + * @param ident the view identifier + * @param info the view metadata + * @return the metadata of the newly created view; may equal {@code info} + * @throws ViewAlreadyExistsException if a view already exists at {@code ident} + * @throws NoSuchNamespaceException if the identifier's namespace does not exist (optional) */ - default View replaceView( - ViewInfo viewInfo, - boolean orCreate) - throws NoSuchViewException, NoSuchNamespaceException { - if (viewExists(viewInfo.ident())) { - dropView(viewInfo.ident()); - } else if (!orCreate) { - throw new NoSuchViewException(viewInfo.ident()); - } - - try { - return createView(viewInfo); - } catch (ViewAlreadyExistsException e) { - throw new RuntimeException("Race condition when creating/replacing view", e); - } - } + ViewInfo createView(Identifier ident, ViewInfo info) + throws ViewAlreadyExistsException, NoSuchNamespaceException; /** - * Apply {@link ViewChange changes} to a view in the catalog. + * Atomically replace an existing view's metadata. *

- * Implementations may reject the requested changes. If any change is rejected, none of the - * changes should be applied to the view. + * Used by {@code ALTER VIEW ... AS}. Implementations should commit the new metadata + * atomically; views carry no data, so a single transactional metastore call (or equivalent) + * is sufficient -- there is no separate staging API. * - * @param ident a view identifier - * @param changes an array of changes to apply to the view - * @return the view altered - * @throws NoSuchViewException If the view doesn't exist or is a table. - * @throws IllegalArgumentException If any change is rejected by the implementation. + * @param ident the view identifier + * @param info the new view metadata + * @return the metadata of the replaced view; may equal {@code info} + * @throws NoSuchViewException if no view exists at {@code ident} */ - View alterView(Identifier ident, ViewChange... changes) - throws NoSuchViewException, IllegalArgumentException; + ViewInfo replaceView(Identifier ident, ViewInfo info) throws NoSuchViewException; /** - * Drop a view in the catalog. + * Create a view if one does not exist at {@code ident}, or atomically replace it if one does. *

- * If the catalog supports tables and contains a table for the identifier and not a view, this - * must not drop the table and must return false. + * Used by {@code CREATE OR REPLACE VIEW}. The default implementation calls + * {@link #replaceView}, falling back to {@link #createView} on + * {@link NoSuchViewException}. The fallback is non-atomic across the two calls (a concurrent + * drop or create can race), so catalogs that can answer the upsert in a single transactional + * call should override this method to collapse to one RPC and to make the swap atomic. * - * @param ident a view identifier - * @return true if a view was deleted, false if no view exists for the identifier + * @param ident the view identifier + * @param info the view metadata + * @return the metadata of the created or replaced view; may equal {@code info} + * @throws ViewAlreadyExistsException if {@code ident} cannot host this view -- either a + * concurrent {@code CREATE VIEW} won the race in the + * default impl's gap between {@link #replaceView} and + * the fallback {@link #createView}, or, in a + * {@link RelationCatalog}, a table sits at {@code ident} + * @throws NoSuchNamespaceException if the identifier's namespace does not exist (optional) */ - boolean dropView(Identifier ident); + default ViewInfo createOrReplaceView(Identifier ident, ViewInfo info) + throws ViewAlreadyExistsException, NoSuchNamespaceException { + try { + return replaceView(ident, info); + } catch (NoSuchViewException e) { + return createView(ident, info); + } + } /** - * Rename a view in the catalog. - *

- * If the catalog supports tables and contains a table with the old identifier, this throws - * {@link NoSuchViewException}. Additionally, if it contains a table with the new identifier, - * this throws {@link ViewAlreadyExistsException}. - *

- * If the catalog does not support view renames between namespaces, it throws - * {@link UnsupportedOperationException}. + * Drop a view. * - * @param oldIdent the view identifier of the existing view to rename - * @param newIdent the new view identifier of the view - * @throws NoSuchViewException If the view to rename doesn't exist or is a table - * @throws ViewAlreadyExistsException If the new view name already exists or is a table - * @throws UnsupportedOperationException If the namespaces of old and new identifiers do not - * match (optional) + * @param ident a view identifier + * @return true if a view was dropped, false otherwise */ - void renameView(Identifier oldIdent, Identifier newIdent) - throws NoSuchViewException, ViewAlreadyExistsException; + boolean dropView(Identifier ident); } diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewChange.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewChange.java deleted file mode 100644 index c94933beed7f6..0000000000000 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewChange.java +++ /dev/null @@ -1,79 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.spark.sql.connector.catalog; - -import org.apache.spark.annotation.DeveloperApi; - -/** - * ViewChange subclasses represent requested changes to a view. - * These are passed to {@link ViewCatalog#alterView}. - */ -@DeveloperApi -public interface ViewChange { - - /** - * Create a ViewChange for setting a table property. - * - * @param property the property name - * @param value the new property value - * @return a ViewChange - */ - static ViewChange setProperty(String property, String value) { - return new SetProperty(property, value); - } - - /** - * Create a ViewChange for removing a table property. - * - * @param property the property name - * @return a ViewChange - */ - static ViewChange removeProperty(String property) { - return new RemoveProperty(property); - } - - final class SetProperty implements ViewChange { - private final String property; - private final String value; - - private SetProperty(String property, String value) { - this.property = property; - this.value = value; - } - - public String property() { - return property; - } - - public String value() { - return value; - } - } - - final class RemoveProperty implements ViewChange { - private final String property; - - private RemoveProperty(String property) { - this.property = property; - } - - public String property() { - return property; - } - } -} diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewInfo.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewInfo.java index b01e133365661..da82de01f8e4d 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewInfo.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewInfo.java @@ -14,168 +14,139 @@ * See the License for the specific language governing permissions and * limitations under the License. */ - package org.apache.spark.sql.connector.catalog; -import org.apache.spark.annotation.DeveloperApi; -import org.apache.spark.sql.types.StructType; - -import javax.annotation.Nonnull; - -import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; import java.util.Map; import java.util.Objects; -import java.util.StringJoiner; + +import org.apache.spark.annotation.Evolving; /** - * A class that holds view information. + * View metadata DTO -- the typed payload returned by {@link ViewCatalog#loadView} and accepted + * by {@link ViewCatalog#createView} / {@link ViewCatalog#replaceView}. Carries the + * view-specific fields that cannot be represented as string table properties: the query text, + * captured creation-time resolution context, captured SQL configs, schema-binding mode, and + * query output column names. Schema and user TBLPROPERTIES are inherited from {@link TableInfo} + * via the typed builder. + *

+ * {@code ViewInfo} extends {@link TableInfo} so that a {@link RelationCatalog} can opt into the + * single-RPC perf path by returning a {@link MetadataOnlyTable} wrapping a {@code ViewInfo} + * from {@link RelationCatalog#loadRelation} for a view identifier. Pure {@link ViewCatalog} + * implementations never see {@code TableInfo}; the typed setters on {@link Builder} cover + * everything they need to construct a {@code ViewInfo}. + * + * @since 4.2.0 */ -@DeveloperApi -public class ViewInfo { - private final Identifier ident; - private final String sql; +@Evolving +public class ViewInfo extends TableInfo { + + private final String queryText; private final String currentCatalog; private final String[] currentNamespace; - private final StructType schema; + private final Map sqlConfigs; + private final String schemaMode; private final String[] queryColumnNames; - private final String[] columnAliases; - private final String[] columnComments; - private final Map properties; - - public ViewInfo( - Identifier ident, - String sql, - String currentCatalog, - String[] currentNamespace, - StructType schema, - String[] queryColumnNames, - String[] columnAliases, - String[] columnComments, - Map properties) { - this.ident = ident; - this.sql = sql; - this.currentCatalog = currentCatalog; - this.currentNamespace = currentNamespace; - this.schema = schema; - this.queryColumnNames = queryColumnNames; - this.columnAliases = columnAliases; - this.columnComments = columnComments; - this.properties = properties; - } - /** - * @return The view identifier - */ - @Nonnull - public Identifier ident() { - return ident; + private ViewInfo(Builder builder) { + super(builder); + this.queryText = Objects.requireNonNull(builder.queryText, "queryText should not be null"); + this.currentCatalog = builder.currentCatalog; + this.currentNamespace = builder.currentNamespace; + this.sqlConfigs = Collections.unmodifiableMap(builder.sqlConfigs); + this.schemaMode = builder.schemaMode; + this.queryColumnNames = builder.queryColumnNames; + // Force PROP_TABLE_TYPE = VIEW so that `properties()` reflects the typed ViewInfo + // classification. Catalogs and generic viewers reading PROP_TABLE_TYPE from the properties + // bag (e.g. TableCatalog.listTableSummaries default impl, DESCRIBE) see "VIEW" without + // requiring authors to remember to call withTableType(VIEW). + properties().put(TableCatalog.PROP_TABLE_TYPE, TableSummary.VIEW_TABLE_TYPE); } - /** - * @return The SQL text that defines the view - */ - @Nonnull - public String sql() { - return sql; - } + /** The SQL text of the view. */ + public String queryText() { return queryText; } /** - * @return The current catalog + * The current catalog at the time the view was created, used to resolve unqualified + * identifiers in {@link #queryText()} at read time. May be {@code null} if the view was + * created with no captured resolution context. */ - @Nonnull - public String currentCatalog() { - return currentCatalog; - } + public String currentCatalog() { return currentCatalog; } /** - * @return The current namespace + * The current namespace at the time the view was created, used alongside + * {@link #currentCatalog()} to resolve unqualified identifiers in {@link #queryText()} at + * read time. Never {@code null}; empty when no namespace was captured. */ - @Nonnull - public String[] currentNamespace() { - return currentNamespace; - } + public String[] currentNamespace() { return currentNamespace; } /** - * @return The view query output schema + * The SQL configs captured at view creation time, applied when parsing and analyzing the + * view body. Keys are unprefixed SQL config names (e.g. {@code spark.sql.ansi.enabled}). */ - @Nonnull - public StructType schema() { - return schema; - } + public Map sqlConfigs() { return sqlConfigs; } /** - * @return The query column names + * The view's schema binding mode. Allowed values match the {@code toString} form of + * {@code org.apache.spark.sql.catalyst.analysis.ViewSchemaMode}: + * {@code BINDING}, {@code COMPENSATION}, {@code TYPE EVOLUTION}, {@code EVOLUTION}. + * May be {@code null} when schema binding is not configured. */ - @Nonnull - public String[] queryColumnNames() { - return queryColumnNames; - } + public String schemaMode() { return schemaMode; } /** - * @return The column aliases + * Output column names of the query that created the view, used to map the query output to + * the view's declared columns during view resolution. Empty for views in {@code EVOLUTION} + * mode, which always use the view's current schema. */ - @Nonnull - public String[] columnAliases() { - return columnAliases; - } + public String[] queryColumnNames() { return queryColumnNames; } + + public static class Builder extends BaseBuilder { + private String queryText; + private String currentCatalog; + private String[] currentNamespace = new String[0]; + private Map sqlConfigs = new HashMap<>(); + private String schemaMode; + private String[] queryColumnNames = new String[0]; + + @Override + protected Builder self() { return this; } + + public Builder withQueryText(String queryText) { + this.queryText = queryText; + return this; + } - /** - * @return The column comments - */ - @Nonnull - public String[] columnComments() { - return columnComments; - } + public Builder withCurrentCatalog(String currentCatalog) { + this.currentCatalog = currentCatalog; + return this; + } - /** - * @return The view properties - */ - @Nonnull - public Map properties() { - return properties; - } + public Builder withCurrentNamespace(String[] currentNamespace) { + this.currentNamespace = currentNamespace == null ? new String[0] : currentNamespace; + return this; + } - @Override - public boolean equals(Object o) { - if (this == o) { - return true; + public Builder withSqlConfigs(Map sqlConfigs) { + this.sqlConfigs = new HashMap<>(sqlConfigs); + return this; } - if (o == null || getClass() != o.getClass()) { - return false; + + public Builder withSchemaMode(String schemaMode) { + this.schemaMode = schemaMode; + return this; } - ViewInfo viewInfo = (ViewInfo) o; - return ident.equals(viewInfo.ident) && sql.equals(viewInfo.sql) && - currentCatalog.equals(viewInfo.currentCatalog) && - Arrays.equals(currentNamespace, viewInfo.currentNamespace) && - schema.equals(viewInfo.schema) && - Arrays.equals(queryColumnNames, viewInfo.queryColumnNames) && - Arrays.equals(columnAliases, viewInfo.columnAliases) && - Arrays.equals(columnComments, viewInfo.columnComments) && - properties.equals(viewInfo.properties); - } - @Override - public int hashCode() { - int result = Objects.hash(ident, sql, currentCatalog, schema, properties); - result = 31 * result + Arrays.hashCode(currentNamespace); - result = 31 * result + Arrays.hashCode(queryColumnNames); - result = 31 * result + Arrays.hashCode(columnAliases); - result = 31 * result + Arrays.hashCode(columnComments); - return result; - } + public Builder withQueryColumnNames(String[] queryColumnNames) { + this.queryColumnNames = queryColumnNames == null ? new String[0] : queryColumnNames; + return this; + } - @Override - public String toString() { - return new StringJoiner(", ", ViewInfo.class.getSimpleName() + "[", "]") - .add("ident=" + ident) - .add("sql='" + sql + "'") - .add("currentCatalog='" + currentCatalog + "'") - .add("currentNamespace=" + Arrays.toString(currentNamespace)) - .add("schema=" + schema) - .add("queryColumnNames=" + Arrays.toString(queryColumnNames)) - .add("columnAliases=" + Arrays.toString(columnAliases)) - .add("columnComments=" + Arrays.toString(columnComments)) - .add("properties=" + properties) - .toString(); + @Override + public ViewInfo build() { + Objects.requireNonNull(columns, "columns should not be null"); + return new ViewInfo(this); + } } } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala index c736c3a8c0ef0..0277b664ff904 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala @@ -50,7 +50,7 @@ import org.apache.spark.sql.catalyst.trees.TreePattern._ import org.apache.spark.sql.catalyst.types.DataTypeUtils import org.apache.spark.sql.catalyst.util.{toPrettySQL, trimTempResolvedColumn, CharVarcharUtils} import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns._ -import org.apache.spark.sql.connector.catalog.{View => _, _} +import org.apache.spark.sql.connector.catalog._ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.TableChange.{After, ColumnPosition} import org.apache.spark.sql.connector.catalog.functions.UnboundFunction @@ -1102,7 +1102,19 @@ class Analyzer( /** * Resolves relations to `ResolvedTable` or `Resolved[Temp/Persistent]View`. This is - * for resolving DDL and misc commands. + * for resolving DDL and misc commands. UnresolvedView callers reject non-view results + * downstream via `expectViewNotTableError`. + * + * When `viewOnly=true`, non-session catalogs that do not implement [[ViewCatalog]] are + * rejected up front with MISSING_CATALOG_ABILITY.VIEWS -- they cannot host views at all, + * so surfacing a downstream "view not found" would hide the real reason. + * + * Lookup order against a non-session catalog: + * 1. If the catalog is a [[RelationCatalog]], [[RelationCatalog.loadRelation]] is called + * once. A returned [[MetadataOnlyTable]] wrapping a [[ViewInfo]] is interpreted as a + * view; other results are tables. + * 2. Otherwise, [[TableCatalog.loadTable]] is tried (when implemented), then + * [[ViewCatalog.loadView]] as the fallback view-resolution path (when implemented). */ private def lookupTableOrView( identifier: Seq[String], @@ -1112,18 +1124,60 @@ class Analyzer( }.orElse { relationResolution.expandIdentifier(identifier) match { case CatalogAndIdentifier(catalog, ident) => - if (viewOnly && !CatalogV2Util.isSessionCatalog(catalog)) { - throw QueryCompilationErrors.catalogOperationNotSupported(catalog, "views") + if (viewOnly && !CatalogV2Util.isSessionCatalog(catalog) && + !catalog.isInstanceOf[ViewCatalog]) { + throw QueryCompilationErrors.missingCatalogViewsAbilityError(catalog) } - CatalogV2Util.loadTable(catalog, ident).map { - case v1Table: V1Table if CatalogV2Util.isSessionCatalog(catalog) && - v1Table.v1Table.tableType == CatalogTableType.VIEW => - val v1Ident = v1Table.catalogTable.identifier - val v2Ident = Identifier.of(v1Ident.database.toArray, v1Ident.identifier) - ResolvedPersistentView( - catalog, v2Ident, v1Table.catalogTable) - case table => - ResolvedTable.create(catalog.asTableCatalog, ident, table) + catalog match { + case mc: RelationCatalog => + // Single-RPC perf path: loadRelation returns a Table for a table or a + // MetadataOnlyTable wrapping a ViewInfo for a view. NoSuchTable means + // neither exists. + try { + Some(mc.loadRelation(ident) match { + case t: MetadataOnlyTable if t.getTableInfo.isInstanceOf[ViewInfo] => + ResolvedPersistentView( + catalog, ident, V1Table.toCatalogTable(catalog, ident, t)) + case table => + ResolvedTable.create(catalog.asTableCatalog, ident, table) + }) + } catch { + case _: NoSuchTableException => None + } + case _ => + // Skip the table-side lookup entirely for view-only catalogs (no + // `TableCatalog` mixin): `CatalogV2Util.loadTable` would call `asTableCatalog` + // and throw MISSING_CATALOG_ABILITY.TABLES, masking the legitimate view- + // resolution path. + val tableResolved: Option[LogicalPlan] = if ( + CatalogV2Util.isSessionCatalog(catalog) || catalog.isInstanceOf[TableCatalog] + ) { + CatalogV2Util.loadTable(catalog, ident).map { + case v1Table: V1Table if CatalogV2Util.isSessionCatalog(catalog) && + v1Table.v1Table.tableType == CatalogTableType.VIEW => + val v1Ident = v1Table.catalogTable.identifier + val v2Ident = Identifier.of(v1Ident.database.toArray, v1Ident.identifier) + ResolvedPersistentView( + catalog, v2Ident, v1Table.catalogTable) + case table => + ResolvedTable.create(catalog.asTableCatalog, ident, table) + } + } else { + None + } + tableResolved.orElse { + catalog match { + case vc: ViewCatalog => + try { + val viewInfo = vc.loadView(ident) + val catalogTable = V1Table.toCatalogTable(catalog, ident, viewInfo) + Some(ResolvedPersistentView(catalog, ident, catalogTable)) + } catch { + case _: NoSuchViewException => None + } + case _ => None + } + } } case _ => None } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala index 67d5b70b30a33..3e8b507e4f6c0 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala @@ -197,7 +197,7 @@ object ApplyDefaultCollation extends Rule[LogicalPlan] { collation = getCollationFromSchemaMetadata(catalog, identifier.namespace()))) case createView@CreateView(ResolvedIdentifier( - catalog: SupportsNamespaces, identifier), _, _, _, _, _, _, _, _, _) + catalog: SupportsNamespaces, identifier), _, _, _, _, _, _, _, _, _, _, _) if createView.collation.isEmpty => val newCreateView = CurrentOrigin.withOrigin(createView.origin) { createView.copy( @@ -209,7 +209,7 @@ object ApplyDefaultCollation extends Rule[LogicalPlan] { // We match against ResolvedPersistentView because temporary views don't have a // schema/catalog. case alterViewAs@AlterViewAs(resolvedPersistentView@ResolvedPersistentView( - catalog: SupportsNamespaces, identifier, _), _, _) + catalog: SupportsNamespaces, identifier, _), _, _, _, _) if resolvedPersistentView.metadata.collation.isEmpty => val newResolvedPersistentView = resolvedPersistentView.copy( metadata = resolvedPersistentView.metadata.copy( diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala index e86248febd2eb..58f832ea6cbdf 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala @@ -23,6 +23,7 @@ import org.apache.spark.internal.Logging import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.SQLConfHelper import org.apache.spark.sql.catalyst.catalog.{ + CatalogTable, CatalogTableType, TemporaryViewRelation, UnresolvedCatalogRelation @@ -36,9 +37,14 @@ import org.apache.spark.sql.connector.catalog.{ ChangelogInfo, Identifier, LookupCatalog, + MetadataOnlyTable, + RelationCatalog, Table, + TableCatalog, V1Table, - V2TableWithV1Fallback + V2TableWithV1Fallback, + ViewCatalog, + ViewInfo } import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.errors.{DataTypeErrorsBase, QueryCompilationErrors} @@ -227,11 +233,60 @@ class RelationResolution( .orElse { val writePrivileges = u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES) val finalOptions = u.clearWritePrivileges.options - val table = CatalogV2Util.loadTable( - catalog, - ident, - finalTimeTravelSpec, - Option(writePrivileges)) + // For a `RelationCatalog` with no time-travel / write privileges, the single-RPC + // `loadRelation` answers both "is there a table?" and "is there a view?" in one + // call. Time-travel and write privileges apply to tables only, so for those the + // lookup falls through to the table-only `loadTable` path below; views are not + // reachable via the v2 fallback in those cases. + // + // Skip the table-side lookup entirely for view-only catalogs (no `TableCatalog` + // mixin): `CatalogV2Util.loadTable` would call `asTableCatalog` and throw + // MISSING_CATALOG_ABILITY.TABLES, masking the legitimate view-resolution path. + val tableOrView: Option[Table] = catalog match { + case mc: RelationCatalog if finalTimeTravelSpec.isEmpty && writePrivileges == null => + try { + Some(mc.loadRelation(ident)) + } catch { + case _: NoSuchTableException => None + } + case _ => + val tableSide: Option[Table] = if ( + CatalogV2Util.isSessionCatalog(catalog) || catalog.isInstanceOf[TableCatalog] + ) { + CatalogV2Util.loadTable( + catalog, + ident, + finalTimeTravelSpec, + Option(writePrivileges)) + } else { + None + } + // Fallback to ViewCatalog for catalogs that host views but where loadTable + // returned None (or was skipped because there's no TableCatalog mixin). + // Time-travel / write privileges only apply to tables, not views, so the + // fallback only fires when both are absent. + tableSide.orElse { + if (finalTimeTravelSpec.isEmpty && writePrivileges == null) { + catalog match { + case vc: ViewCatalog => + try { + Some(new MetadataOnlyTable(vc.loadView(ident), ident.toString)) + } catch { + case _: NoSuchViewException => None + } + case _ => None + } + } else { + None + } + } + } + // `table` is `tableOrView` filtered to tables only -- used for cache lookup since + // we don't share-cache views. + val table: Option[Table] = tableOrView.filter { + case t: MetadataOnlyTable if t.getTableInfo.isInstanceOf[ViewInfo] => false + case _ => true + } val sharedRelationCacheMatch = for { t <- table @@ -249,7 +304,7 @@ class RelationResolution( val loaded = createRelation( catalog, ident, - table, + tableOrView, finalOptions, u.isStreaming, finalTimeTravelSpec) @@ -314,6 +369,22 @@ class RelationResolution( options: CaseInsensitiveStringMap, isStreaming: Boolean, timeTravelSpec: Option[TimeTravelSpec]): Option[LogicalPlan] = { + def createDataSourceV1Scan(v1Table: CatalogTable): LogicalPlan = { + if (isStreaming) { + if (v1Table.tableType == CatalogTableType.VIEW) { + throw QueryCompilationErrors.permanentViewNotSupportedByStreamingReadingAPIError( + ident.quoted + ) + } + SubqueryAlias( + v1Table.fullIdent, + UnresolvedCatalogRelation(v1Table, options, isStreaming = true) + ) + } else { + v1SessionCatalog.getRelation(v1Table, options) + } + } + table.map { // To utilize this code path to execute V1 commands, e.g. INSERT, // either it must be session catalog, or tracksPartitionsInCatalog @@ -324,19 +395,13 @@ class RelationResolution( case v1Table: V1Table if CatalogV2Util.isSessionCatalog(catalog) || !v1Table.catalogTable.tracksPartitionsInCatalog => - if (isStreaming) { - if (v1Table.v1Table.tableType == CatalogTableType.VIEW) { - throw QueryCompilationErrors.permanentViewNotSupportedByStreamingReadingAPIError( - ident.quoted - ) - } - SubqueryAlias( - catalog.name +: ident.asMultipartIdentifier, - UnresolvedCatalogRelation(v1Table.v1Table, options, isStreaming = true) - ) - } else { - v1SessionCatalog.getRelation(v1Table.v1Table, options) - } + createDataSourceV1Scan(v1Table.v1Table) + + // MetadataOnlyTable is a sentinel meaning "interpret via v1", so unlike the V1Table + // case above we apply no session-catalog / tracksPartitionsInCatalog guard -- any catalog + // returning MetadataOnlyTable has opted into v1 read semantics. + case t: MetadataOnlyTable => + createDataSourceV1Scan(V1Table.toCatalogTable(catalog, ident, t)) case table => if (isStreaming) { diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ViewResolution.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ViewResolution.scala index faa3b9081cbfd..b0f0ef3b092c1 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ViewResolution.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ViewResolution.scala @@ -38,7 +38,7 @@ object ViewResolution { val maxNestedViewDepth = AnalysisContext.get.maxNestedViewDepth if (nestedViewDepth > maxNestedViewDepth) { throw QueryCompilationErrors.viewDepthExceedsMaxResolutionDepthError( - view.desc.identifier, + view.desc.fullIdent, maxNestedViewDepth, view ) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ViewResolver.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ViewResolver.scala index 992f065ef3aa2..a224e521b548b 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ViewResolver.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ViewResolver.scala @@ -193,7 +193,7 @@ case class ViewResolutionContext( def validate(unresolvedView: View): Unit = { if (nestedViewDepth > maxNestedViewDepth) { throw QueryCompilationErrors.viewDepthExceedsMaxResolutionDepthError( - unresolvedView.desc.identifier, + unresolvedView.desc.fullIdent, maxNestedViewDepth, unresolvedView ) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala index ff4a135b7d044..af398eb8527e9 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala @@ -1054,10 +1054,15 @@ class SessionCatalog( def getRelation( metadata: CatalogTable, options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty()): LogicalPlan = { - val qualifiedIdent = qualifyIdentifier(metadata.identifier) - val db = qualifiedIdent.database.get - val table = qualifiedIdent.table - val multiParts = Seq(CatalogManager.SESSION_CATALOG_NAME, db, table) + // Prefer `multipartIdentifier` (set by non-session v2 catalogs via `V1Table.toCatalogTable`) + // so the SubqueryAlias qualifier reflects the real catalog + multi-part namespace. + // Fall back to the historical 3-part form for v1 session-catalog tables -- we intentionally + // always include `SESSION_CATALOG_NAME` here and ignore + // `LEGACY_NON_IDENTIFIER_OUTPUT_CATALOG_NAME` to preserve pre-v2-MetadataOnlyTable behavior. + val multiParts = metadata.multipartIdentifier.getOrElse { + val qualifiedIdent = qualifyIdentifier(metadata.identifier) + Seq(CatalogManager.SESSION_CATALOG_NAME, qualifiedIdent.database.get, qualifiedIdent.table) + } if (CatalogTable.isMetricView(metadata)) { parseMetricViewDefinition(metadata) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala index 1cc4f7bcc3d29..981b2ac96a37a 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala @@ -445,11 +445,22 @@ case class CatalogTable( tracksPartitionsInCatalog: Boolean = false, schemaPreservesCase: Boolean = true, ignoredProperties: Map[String, String] = Map.empty, - viewOriginalText: Option[String] = None) + viewOriginalText: Option[String] = None, + // Multi-part identifier [catalog, namespace..., name] for tables synthesized from a v2 + // `MetadataOnlyTable` whose namespace has more than one part -- the v1 `identifier: + // TableIdentifier` (single-string database) cannot carry that losslessly. `None` for + // v1-native tables; callers should use `fullIdent` which falls back to `identifier.nameParts`. + multipartIdentifier: Option[Seq[String]] = None) extends MetadataMapSupport { import CatalogTable._ + /** + * The fully-qualified multi-part identifier. Prefers `multipartIdentifier` when set (v2-sourced + * tables with multi-level namespaces); otherwise reconstructs from `identifier.nameParts`. + */ + def fullIdent: Seq[String] = multipartIdentifier.getOrElse(identifier.nameParts) + /** * schema of this table's partition columns */ @@ -544,20 +555,7 @@ case class CatalogTable( * Return the schema binding mode. Defaults to SchemaBinding if not a view or an older * version, unless the viewSchemaBindingMode config is set to false */ - def viewSchemaMode: ViewSchemaMode = { - if (!SQLConf.get.viewSchemaBindingEnabled) { - SchemaUnsupported - } else { - val schemaMode = properties.getOrElse(VIEW_SCHEMA_MODE, SchemaBinding.toString) - schemaMode match { - case SchemaBinding.toString => SchemaBinding - case SchemaEvolution.toString => SchemaEvolution - case SchemaTypeEvolution.toString => SchemaTypeEvolution - case SchemaCompensation.toString => SchemaCompensation - case other => throw SparkException.internalError("Unexpected ViewSchemaMode") - } - } - } + def viewSchemaMode: ViewSchemaMode = CatalogTable.viewSchemaModeFromProperties(properties) /** * Return temporary view names the current view was referred. should be empty if the @@ -789,6 +787,26 @@ object CatalogTable { val PROP_CLUSTERING_COLUMNS: String = "clusteringColumns" + /** + * Decode the view schema binding mode from a properties map. Shared between + * [[CatalogTable.viewSchemaMode]] and the v2 ALTER VIEW path which reads the mode directly + * from the existing view's [[TableInfo]] properties without materializing a full CatalogTable. + */ + def viewSchemaModeFromProperties(properties: Map[String, String]): ViewSchemaMode = { + if (!SQLConf.get.viewSchemaBindingEnabled) { + SchemaUnsupported + } else { + val schemaMode = properties.getOrElse(VIEW_SCHEMA_MODE, SchemaBinding.toString) + schemaMode match { + case SchemaBinding.toString => SchemaBinding + case SchemaEvolution.toString => SchemaEvolution + case SchemaTypeEvolution.toString => SchemaTypeEvolution + case SchemaCompensation.toString => SchemaCompensation + case _ => throw SparkException.internalError("Unexpected ViewSchemaMode") + } + } + } + def splitLargeTableProp( key: String, value: String, diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala index f63de8d1e4656..0eded2d9dbdf9 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala @@ -1363,8 +1363,12 @@ case class ShowTablePartition( /** * The logical plan of the SHOW VIEWS command. * - * Notes: v2 catalogs do not support views API yet, the command will fallback to - * v1 ShowViewsCommand during ResolveSessionCatalog. + * Session-catalog targets fall back to v1 `ShowViewsCommand` via `ResolveSessionCatalog`. + * v2 [[org.apache.spark.sql.connector.catalog.ViewCatalog]] catalogs are handled in + * `DataSourceV2Strategy` (enumerates via + * [[org.apache.spark.sql.connector.catalog.ViewCatalog#listViews]]). Non-ViewCatalog v2 + * catalogs are rejected up front in `ResolveSessionCatalog` with + * `MISSING_CATALOG_ABILITY.VIEWS`. */ case class ShowViews( namespace: LogicalPlan, @@ -1714,19 +1718,42 @@ case class RepairTable( /** * The logical plan of the ALTER VIEW ... AS command. + * + * Extends [[AnalysisOnlyCommand]] so [[Analyzer.HandleSpecialCommand]] captures + * `referredTempFunctions` from [[AnalysisContext]]; this list is needed by + * [[CheckViewReferences]] and by the v2 execs when the target is a non-session catalog. + * Session-catalog targets are still rewritten to [[AlterViewAsCommand]] by + * `ResolveSessionCatalog` and the captured value is dropped there (the v1 command re-captures). */ case class AlterViewAs( child: LogicalPlan, originalText: String, - query: LogicalPlan) extends BinaryCommand with CTEInChildren { - override def left: LogicalPlan = child - override def right: LogicalPlan = query + query: LogicalPlan, + isAnalyzed: Boolean = false, + referredTempFunctions: Seq[String] = Seq.empty) + extends Command with AnalysisOnlyCommand with CTEInChildren { + + override def childrenToAnalyze: Seq[LogicalPlan] = Seq(child, query) + + override def markAsAnalyzed(analysisContext: AnalysisContext): LogicalPlan = copy( + isAnalyzed = true, + referredTempFunctions = analysisContext.referredTempFunctionNames.toSeq) + override protected def withNewChildrenInternal( - newLeft: LogicalPlan, newRight: LogicalPlan): LogicalPlan = - copy(child = newLeft, query = newRight) + newChildren: IndexedSeq[LogicalPlan]): LogicalPlan = { + assert(!isAnalyzed) + newChildren match { + case Seq(newChild, newQuery) => + copy(child = newChild, query = newQuery) + case others => + throw new SparkIllegalArgumentException( + errorClass = "_LEGACY_ERROR_TEMP_3218", + messageParameters = Map("others" -> others.toString())) + } + } override def withCTEDefs(cteDefs: Seq[CTERelationDef]): LogicalPlan = { - withNewChildren(Seq(child, WithCTE(query, cteDefs))) + copy(query = WithCTE(query, cteDefs)) } } @@ -1743,6 +1770,11 @@ case class AlterViewSchemaBinding( /** * The logical plan of the CREATE VIEW ... command. + * + * Extends [[AnalysisOnlyCommand]] so that [[Analyzer.HandleSpecialCommand]] captures + * `referredTempFunctions` from the [[AnalysisContext]] after the child query is analyzed; + * this list is needed for `verifyTemporaryObjectsNotExists`-style checks on downstream + * execution paths. */ case class CreateView( child: LogicalPlan, @@ -1754,15 +1786,32 @@ case class CreateView( query: LogicalPlan, allowExisting: Boolean, replace: Boolean, - viewSchemaMode: ViewSchemaMode) extends BinaryCommand with CTEInChildren { - override def left: LogicalPlan = child - override def right: LogicalPlan = query + viewSchemaMode: ViewSchemaMode, + isAnalyzed: Boolean = false, + referredTempFunctions: Seq[String] = Seq.empty) + extends Command with AnalysisOnlyCommand with CTEInChildren { + + override def childrenToAnalyze: Seq[LogicalPlan] = Seq(child, query) + + override def markAsAnalyzed(analysisContext: AnalysisContext): LogicalPlan = copy( + isAnalyzed = true, + referredTempFunctions = analysisContext.referredTempFunctionNames.toSeq) + override protected def withNewChildrenInternal( - newLeft: LogicalPlan, newRight: LogicalPlan): LogicalPlan = - copy(child = newLeft, query = newRight) + newChildren: IndexedSeq[LogicalPlan]): LogicalPlan = { + assert(!isAnalyzed) + newChildren match { + case Seq(newChild, newQuery) => + copy(child = newChild, query = newQuery) + case others => + throw new SparkIllegalArgumentException( + errorClass = "_LEGACY_ERROR_TEMP_3218", + messageParameters = Map("others" -> others.toString())) + } + } override def withCTEDefs(cteDefs: Seq[CTERelationDef]): LogicalPlan = { - withNewChildren(Seq(child, WithCTE(query, cteDefs))) + copy(query = WithCTE(query, cteDefs)) } } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala index cf6052009c927..a5f1ca7f1d289 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala @@ -171,6 +171,15 @@ private[sql] object CatalogV2Implicits { throw QueryCompilationErrors.requiresSinglePartNamespaceError(asMultipartIdentifier) } + // Build a v1 TableIdentifier for display / error-rendering purposes. Collapses a + // multi-part namespace to its last segment (v1 TableIdentifier has a single-string + // database field). Callers that need a lossless multi-part form should build a + // Seq[String] from toQualifiedNameParts instead. + def asLegacyTableIdentifier(catalogName: String): TableIdentifier = TableIdentifier( + table = ident.name(), + database = ident.namespace().lastOption, + catalog = Some(catalogName)) + /** * Tries to convert catalog identifier to the table identifier. Table identifier does not * support multiple namespaces (nested namespaces), so if identifier contains nested namespace, diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/Catalogs.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/Catalogs.scala index e6c70fdabb159..03addeb170697 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/Catalogs.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/Catalogs.scala @@ -64,6 +64,7 @@ private[sql] object Catalogs { } val plugin = pluginClass.getDeclaredConstructor().newInstance().asInstanceOf[CatalogPlugin] plugin.initialize(name, catalogOptions(name, conf)) + validateRelationCatalog(name, plugin) plugin } catch { case e: ClassNotFoundException => @@ -106,4 +107,22 @@ private[sql] object Catalogs { } new CaseInsensitiveStringMap(options) } + + /** + * Reject catalogs that implement both [[TableCatalog]] and [[ViewCatalog]] without + * extending [[RelationCatalog]]. The combined case has cross-cutting rules (single namespace, + * cross-type collision rejection, perf opt-ins) that live on [[RelationCatalog]]; implementing + * the two interfaces directly would skip that contract. + */ + private def validateRelationCatalog(name: String, plugin: CatalogPlugin): Unit = { + if (plugin.isInstanceOf[TableCatalog] && plugin.isInstanceOf[ViewCatalog] && + !plugin.isInstanceOf[RelationCatalog]) { + throw new IllegalArgumentException( + s"Catalog '$name' (${plugin.getClass.getName}) implements both TableCatalog and " + + s"ViewCatalog directly. Catalogs that expose both tables and views must implement " + + s"RelationCatalog instead, which centralizes the cross-cutting rules (shared " + + s"identifier namespace, cross-type collision rejection, single-RPC perf entry " + + s"points).") + } + } } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/V1Table.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/V1Table.scala index eee6ddf3e58fd..079b2639aa2b9 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/V1Table.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/V1Table.scala @@ -22,8 +22,8 @@ import java.util import scala.collection.mutable import scala.jdk.CollectionConverters._ -import org.apache.spark.sql.catalyst.catalog.{CatalogTable, CatalogTableType, CatalogUtils} -import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.TableIdentifierHelper +import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType, CatalogUtils, ClusterBySpec} +import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.V1Table.addV2TableProperties import org.apache.spark.sql.connector.expressions.{LogicalExpressions, Transform} import org.apache.spark.sql.types.StructType @@ -49,7 +49,6 @@ private[sql] case class V1Table(v1Table: CatalogTable) extends Table { override lazy val schema: StructType = v1Table.schema override lazy val partitioning: Array[Transform] = { - import CatalogV2Implicits._ val partitions = new mutable.ArrayBuffer[Transform]() v1Table.partitionColumnNames.foreach { col => @@ -109,6 +108,108 @@ private[sql] object V1Table { case _ => None } } + + def toCatalogTable( + catalog: CatalogPlugin, + ident: Identifier, + t: MetadataOnlyTable): CatalogTable = t.getTableInfo match { + case viewInfo: ViewInfo => toCatalogTable(catalog, ident, viewInfo) + case tableInfo => toCatalogTable(catalog, ident, tableInfo) + } + + private def toCatalogTable( + catalog: CatalogPlugin, + ident: Identifier, + info: TableInfo): CatalogTable = { + val props = info.properties.asScala.toMap + // PROP_TABLE_TYPE is advisory on the v2 side: it may be absent or carry a value that has no + // v1 mapping (e.g. TableSummary.FOREIGN_TABLE_TYPE). v1 only has EXTERNAL/MANAGED, so + // anything other than the explicit MANAGED mapping falls back to EXTERNAL for the v1 + // representation -- the same default v1 uses when the value is missing. VIEW is reached + // only through the ViewInfo branch above. + val tableType = props.get(TableCatalog.PROP_TABLE_TYPE) match { + case Some(TableSummary.MANAGED_TABLE_TYPE) => CatalogTableType.MANAGED + case _ => CatalogTableType.EXTERNAL + } + // Reserved keys are promoted to first-class CatalogTable fields; strip them from the + // user-visible properties map so they're not double-persisted or leaked into the serde bag. + val userProps = props -- CatalogV2Util.TABLE_RESERVED_PROPERTIES + val (serdeProps, tableProps) = userProps.toSeq + .partition(_._1.startsWith(TableCatalog.OPTION_PREFIX)) + val tablePropsMap = tableProps.toMap + val (partCols, bucketSpec, clusterBySpec) = info.partitions.toSeq.convertTransforms + CatalogTable( + // `asLegacyTableIdentifier` collapses multi-part namespaces to their last segment (v1 + // limitation). We record the full multi-part form in `multipartIdentifier` below; + // callers needing the real fully-qualified name should read `CatalogTable.fullIdent`. + identifier = ident.asLegacyTableIdentifier(catalog.name()), + tableType = tableType, + storage = CatalogStorageFormat.empty.copy( + locationUri = props.get(TableCatalog.PROP_LOCATION).map(CatalogUtils.stringToURI), + // v2 table properties should be put into the serde properties as well in case + // they contain data source options. + properties = tablePropsMap ++ serdeProps.map { + case (k, v) => k.drop(TableCatalog.OPTION_PREFIX.length) -> v + } + ), + schema = CatalogV2Util.v2ColumnsToStructType(info.columns), + provider = props.get(TableCatalog.PROP_PROVIDER), + partitionColumnNames = partCols, + bucketSpec = bucketSpec, + owner = props.getOrElse(TableCatalog.PROP_OWNER, ""), + comment = props.get(TableCatalog.PROP_COMMENT), + collation = props.get(TableCatalog.PROP_COLLATION), + properties = tablePropsMap ++ + clusterBySpec.map(ClusterBySpec.toPropertyWithoutValidation), + multipartIdentifier = Some(catalog.name() +: ident.asMultipartIdentifier) + ) + } + + def toCatalogTable( + catalog: CatalogPlugin, + ident: Identifier, + info: ViewInfo): CatalogTable = { + val props = info.properties.asScala.toMap + val userProps = props -- CatalogV2Util.TABLE_RESERVED_PROPERTIES + // Serde/OPTION properties only apply to data-source tables; views' user properties are a + // plain TBLPROPERTIES bag. + val tablePropsMap = userProps + val viewContextProps = if (info.currentCatalog != null && info.currentCatalog.nonEmpty) { + CatalogTable.catalogAndNamespaceToProps( + info.currentCatalog, info.currentNamespace.toSeq) + } else { + Map.empty[String, String] + } + val sqlConfigProps = info.sqlConfigs.asScala.map { + case (k, v) => s"${CatalogTable.VIEW_SQL_CONFIG_PREFIX}$k" -> v + }.toMap + val queryOutputProps = if (info.queryColumnNames.isEmpty) { + Map.empty[String, String] + } else { + val numCols = info.queryColumnNames.length + val perColProps = info.queryColumnNames.zipWithIndex.map { case (name, idx) => + s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}$idx" -> name + }.toMap + perColProps + (CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> numCols.toString) + } + val schemaModeProps = Option(info.schemaMode) + .map(m => Map(CatalogTable.VIEW_SCHEMA_MODE -> m)) + .getOrElse(Map.empty) + CatalogTable( + identifier = ident.asLegacyTableIdentifier(catalog.name()), + tableType = CatalogTableType.VIEW, + storage = CatalogStorageFormat.empty, + schema = CatalogV2Util.v2ColumnsToStructType(info.columns), + owner = props.getOrElse(TableCatalog.PROP_OWNER, ""), + viewText = Some(info.queryText), + viewOriginalText = Some(info.queryText), + comment = props.get(TableCatalog.PROP_COMMENT), + collation = props.get(TableCatalog.PROP_COLLATION), + properties = tablePropsMap ++ viewContextProps ++ sqlConfigProps ++ + queryOutputProps ++ schemaModeProps, + multipartIdentifier = Some(catalog.name() +: ident.asMultipartIdentifier) + ) + } } /** diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala index 512d6f6305266..02e9f188e0fa4 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala @@ -567,11 +567,11 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat } def viewDepthExceedsMaxResolutionDepthError( - identifier: TableIdentifier, maxNestedDepth: Int, t: TreeNode[_]): Throwable = { + viewNameParts: Seq[String], maxNestedDepth: Int, t: TreeNode[_]): Throwable = { new AnalysisException( errorClass = "VIEW_EXCEED_MAX_NESTED_DEPTH", messageParameters = Map( - "viewName" -> toSQLId(identifier.nameParts), + "viewName" -> toSQLId(viewNameParts), "maxNestedDepth" -> maxNestedDepth.toString), origin = t.origin) } @@ -3353,25 +3353,25 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat } def cannotCreateViewTooManyColumnsError( - viewIdent: TableIdentifier, + viewNameParts: Seq[String], expected: Seq[String], query: LogicalPlan): Throwable = { new AnalysisException( errorClass = "CREATE_VIEW_COLUMN_ARITY_MISMATCH.TOO_MANY_DATA_COLUMNS", messageParameters = Map( - "viewName" -> toSQLId(viewIdent.nameParts), + "viewName" -> toSQLId(viewNameParts), "viewColumns" -> expected.map(c => toSQLId(c)).mkString(", "), "dataColumns" -> query.output.map(c => toSQLId(c.name)).mkString(", "))) } def cannotCreateViewNotEnoughColumnsError( - viewIdent: TableIdentifier, + viewNameParts: Seq[String], expected: Seq[String], query: LogicalPlan): Throwable = { new AnalysisException( errorClass = "CREATE_VIEW_COLUMN_ARITY_MISMATCH.NOT_ENOUGH_DATA_COLUMNS", messageParameters = Map( - "viewName" -> toSQLId(viewIdent.nameParts), + "viewName" -> toSQLId(viewNameParts), "viewColumns" -> expected.map(c => toSQLId(c)).mkString(", "), "dataColumns" -> query.output.map(c => toSQLId(c.name)).mkString(", "))) } @@ -3383,12 +3383,12 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat } def unsupportedCreateOrReplaceViewOnTableError( - name: TableIdentifier, replace: Boolean): Throwable = { + nameParts: Seq[String], replace: Boolean): Throwable = { if (replace) { new AnalysisException( errorClass = "EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE", messageParameters = Map( - "tableName" -> toSQLId(name.nameParts), + "tableName" -> toSQLId(nameParts), "operation" -> "CREATE OR REPLACE VIEW" ) ) @@ -3396,16 +3396,16 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat new AnalysisException( errorClass = "TABLE_OR_VIEW_ALREADY_EXISTS", messageParameters = Map( - "relationName" -> toSQLId(name.nameParts) + "relationName" -> toSQLId(nameParts) ) ) } } - def viewAlreadyExistsError(name: TableIdentifier): Throwable = { + def viewAlreadyExistsError(nameParts: Seq[String]): Throwable = { new AnalysisException( errorClass = "TABLE_OR_VIEW_ALREADY_EXISTS", - messageParameters = Map("relationName" -> name.toString)) + messageParameters = Map("relationName" -> toSQLId(nameParts))) } def createPersistedViewFromDatasetAPINotAllowedError(): Throwable = { @@ -3415,57 +3415,57 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat } def recursiveViewDetectedError( - viewIdent: TableIdentifier, - newPath: Seq[TableIdentifier]): Throwable = { + viewIdent: Seq[String], + newPath: Seq[Seq[String]]): Throwable = { new AnalysisException( errorClass = "RECURSIVE_VIEW", messageParameters = Map( - "viewIdent" -> toSQLId(viewIdent.nameParts), - "newPath" -> newPath.map(p => toSQLId(p.nameParts)).mkString(" -> "))) + "viewIdent" -> toSQLId(viewIdent), + "newPath" -> newPath.map(toSQLId).mkString(" -> "))) } def notAllowedToCreatePermanentViewWithoutAssigningAliasForExpressionError( - name: TableIdentifier, + viewNameParts: Seq[String], attr: Attribute): Throwable = { new AnalysisException( errorClass = "CREATE_PERMANENT_VIEW_WITHOUT_ALIAS", messageParameters = Map( - "name" -> toSQLId(name.nameParts), + "name" -> toSQLId(viewNameParts), "attr" -> toSQLExpr(attr))) } def notAllowedToCreatePermanentViewByReferencingTempViewError( - name: TableIdentifier, - nameParts: String): Throwable = { + viewNameParts: Seq[String], + tempViewNameParts: String): Throwable = { new AnalysisException( errorClass = "INVALID_TEMP_OBJ_REFERENCE", messageParameters = Map( "obj" -> "VIEW", - "objName" -> toSQLId(name.nameParts), + "objName" -> toSQLId(viewNameParts), "tempObj" -> "VIEW", - "tempObjName" -> toSQLId(nameParts))) + "tempObjName" -> toSQLId(tempViewNameParts))) } def notAllowedToCreatePermanentViewByReferencingTempFuncError( - name: TableIdentifier, + viewNameParts: Seq[String], funcName: String): Throwable = { new AnalysisException( errorClass = "INVALID_TEMP_OBJ_REFERENCE", messageParameters = Map( "obj" -> "VIEW", - "objName" -> toSQLId(name.nameParts), + "objName" -> toSQLId(viewNameParts), "tempObj" -> "FUNCTION", "tempObjName" -> toSQLId(funcName))) } def notAllowedToCreatePermanentViewByReferencingTempVarError( - nameParts: Seq[String], + viewNameParts: Seq[String], varName: Seq[String]): Throwable = { new AnalysisException( errorClass = "INVALID_TEMP_OBJ_REFERENCE", messageParameters = Map( "obj" -> "VIEW", - "objName" -> toSQLId(nameParts), + "objName" -> toSQLId(viewNameParts), "tempObj" -> "VARIABLE", "tempObjName" -> toSQLId(varName))) } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala index e2bfaef1e7002..94523dd313b43 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala @@ -28,7 +28,7 @@ import org.apache.spark.sql.catalyst.plans.logical._ import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.catalyst.util.{quoteIfNeeded, toPrettySQL, CharVarcharUtils, ResolveDefaultColumns => DefaultCols} import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns._ -import org.apache.spark.sql.connector.catalog.{CatalogExtension, CatalogManager, CatalogPlugin, CatalogV2Util, LookupCatalog, SupportsNamespaces, V1Table} +import org.apache.spark.sql.connector.catalog.{CatalogExtension, CatalogManager, CatalogPlugin, CatalogV2Util, LookupCatalog, SupportsNamespaces, V1Table, ViewCatalog} import org.apache.spark.sql.connector.expressions.Transform import org.apache.spark.sql.errors.{QueryCompilationErrors, QueryExecutionErrors} import org.apache.spark.sql.execution.command._ @@ -327,11 +327,16 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager) case DropView(DropViewInSessionCatalog(ident), ifExists) => DropTableCommand(ident, ifExists, isView = true, purge = false) - case DropView(r @ ResolvedIdentifier(catalog, ident), ifExists) => + // ViewCatalog catalogs fall through to `DataSourceV2Strategy`, which routes DROP VIEW to + // `ViewCatalog.dropView`. Other non-session catalogs get `MISSING_CATALOG_ABILITY.VIEWS`, + // matching the error raised from `CheckViewReferences` for CREATE/ALTER VIEW and from the + // analyzer gate on UnresolvedView. + case DropView(r @ ResolvedIdentifier(catalog, ident), ifExists) + if !catalog.isInstanceOf[ViewCatalog] => if (catalog == FakeSystemCatalog) { DropTempViewCommand(ident, ifExists) } else { - throw QueryCompilationErrors.catalogOperationNotSupported(catalog, "views") + throw QueryCompilationErrors.missingCatalogViewsAbilityError(catalog) } case c @ CreateNamespace(DatabaseNameInSessionCatalog(name), _, _) if conf.useV1Command => @@ -517,14 +522,21 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager) location) => AlterTableSetLocationCommand(ident, Some(partitionSpec), location) - case AlterViewAs(ResolvedViewIdentifier(ident), originalText, query) => + // The final `_, _` are AlterViewAs.isAnalyzed and referredTempFunctions. We drop both: + // AlterViewAsCommand is a separate AnalysisOnlyCommand and gets its own markAsAnalyzed pass + // from HandleSpecialCommand after this rewrite. + case AlterViewAs(ResolvedViewIdentifier(ident), originalText, query, _, _) => AlterViewAsCommand(ident, originalText, query) case AlterViewSchemaBinding(ResolvedViewIdentifier(ident), viewSchemaMode) => AlterViewSchemaBindingCommand(ident, viewSchemaMode) + // The final `_, _` are CreateView.isAnalyzed and referredTempFunctions. We drop both: + // CreateViewCommand is a separate AnalysisOnlyCommand and gets its own markAsAnalyzed pass + // from HandleSpecialCommand after this rewrite. case CreateView(CreateViewInSessionCatalog(ident), userSpecifiedColumns, comment, - collation, properties, originalText, child, allowExisting, replace, viewSchemaMode) => + collation, properties, originalText, query, allowExisting, replace, viewSchemaMode, + _, _) => CreateViewCommand( name = ident, userSpecifiedColumns = userSpecifiedColumns, @@ -532,16 +544,17 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager) collation = collation, properties = properties, originalText = originalText, - plan = child, + plan = query, allowExisting = allowExisting, replace = replace, viewType = PersistedView, viewSchemaMode = viewSchemaMode) - case CreateView(ResolvedIdentifier(catalog, _), _, _, _, _, _, _, _, _, _) => - throw QueryCompilationErrors.missingCatalogViewsAbilityError(catalog) - - case ShowViews(ns: ResolvedNamespace, pattern, output) => + // ViewCatalog catalogs are handled by the v2 strategy (enumerates via listViews); we skip + // the match here so the plan flows through unchanged. Only non-session, non-ViewCatalog + // catalogs hit the MISSING_CATALOG_ABILITY.VIEWS rejection. + case ShowViews(ns: ResolvedNamespace, pattern, output) + if !ns.catalog.isInstanceOf[ViewCatalog] => ns match { case ResolvedDatabaseInSessionCatalog(db) => ShowViewsCommand(db, pattern, output) case _ => @@ -772,9 +785,14 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager) } object ResolvedViewIdentifier { + // Only matches session-catalog persistent views. Non-session-catalog persistent views + // (produced for `MetadataOnlyTable`) fall through; `AlterViewAs` is picked up by the v2 + // strategy, and the remaining view DDL / inspection plans (SET/UNSET TBLPROPERTIES, + // ALTER VIEW ... WITH SCHEMA, RENAME TO, SHOW CREATE TABLE, SHOW TBLPROPERTIES, SHOW + // COLUMNS, DESCRIBE [COLUMN]) are rejected with `UNSUPPORTED_FEATURE.TABLE_OPERATION` by + // dedicated v2 strategy cases -- tracked for a follow-up PR (SPARK-52729). def unapply(resolved: LogicalPlan): Option[TableIdentifier] = resolved match { - case ResolvedPersistentView(catalog, ident, _) => - assert(isSessionCatalog(catalog)) + case ResolvedPersistentView(catalog, ident, _) if isSessionCatalog(catalog) => Some(ident.asTableIdentifier.copy(catalog = Some(catalog.name))) case ResolvedTempView(ident, _) => @@ -938,4 +956,5 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager) SQLConf.get.getConf(SQLConf.V2_SESSION_CATALOG_IMPLEMENTATION) == "builtin" || catalog.isInstanceOf[CatalogExtension]) } + } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/metricViewCommands.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/metricViewCommands.scala index 8c21a908ddf32..623685f6c20a7 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/metricViewCommands.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/metricViewCommands.scala @@ -51,10 +51,10 @@ case class CreateMetricViewCommand( if (userSpecifiedColumns.nonEmpty) { if (userSpecifiedColumns.length > analyzed.output.length) { throw QueryCompilationErrors.cannotCreateViewNotEnoughColumnsError( - name, userSpecifiedColumns.map(_._1), analyzed) + name.nameParts, userSpecifiedColumns.map(_._1), analyzed) } else if (userSpecifiedColumns.length < analyzed.output.length) { throw QueryCompilationErrors.cannotCreateViewTooManyColumnsError( - name, userSpecifiedColumns.map(_._1), analyzed) + name.nameParts, userSpecifiedColumns.map(_._1), analyzed) } } catalog.createTable( @@ -90,7 +90,8 @@ object MetricViewHelper { val metricViewNode = MetricViewPlanner.planWrite( tableMeta, viewText, session.sessionState.sqlParser) val analyzed = analyzer.executeAndCheck(metricViewNode, new QueryPlanningTracker) - ViewHelper.verifyTemporaryObjectsNotExists(isTemporary = false, name, analyzed, Seq.empty) + ViewHelper.verifyTemporaryObjectsNotExists( + isTemporary = false, name.nameParts, analyzed, Seq.empty) analyzed } } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala index 895c39dd83976..994c7836f9dd1 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala @@ -26,15 +26,15 @@ import org.apache.spark.SparkException import org.apache.spark.internal.Logging import org.apache.spark.sql.{Row, SparkSession} import org.apache.spark.sql.catalyst.{CapturesConfig, SQLConfHelper, TableIdentifier} -import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, GlobalTempView, LocalTempView, SchemaEvolution, SchemaUnsupported, ViewSchemaMode, ViewType} +import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, GlobalTempView, LocalTempView, ResolvedIdentifier, ResolvedPersistentView, SchemaEvolution, SchemaUnsupported, ViewSchemaMode, ViewType} import org.apache.spark.sql.catalyst.analysis.V2TableReference import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType, TemporaryViewRelation} import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, SubqueryExpression, VariableReference} -import org.apache.spark.sql.catalyst.plans.logical.{AnalysisOnlyCommand, CreateTempView, CTEInChildren, CTERelationDef, LogicalPlan, Project, View, WithCTE} +import org.apache.spark.sql.catalyst.plans.logical.{AlterViewAs, AnalysisOnlyCommand, CreateTempView, CreateView, CTEInChildren, CTERelationDef, LogicalPlan, Project, View, WithCTE} import org.apache.spark.sql.catalyst.util.CharVarcharUtils import org.apache.spark.sql.classic.ClassicConversions.castToImpl -import org.apache.spark.sql.connector.catalog.CatalogManager -import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.NamespaceHelper +import org.apache.spark.sql.connector.catalog.{CatalogManager, CatalogPlugin, Identifier, ViewCatalog} +import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.{IdentifierHelper, NamespaceHelper} import org.apache.spark.sql.errors.QueryCompilationErrors import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation import org.apache.spark.sql.internal.StaticSQLConf @@ -47,6 +47,12 @@ import org.apache.spark.util.ArrayImplicits._ * properties(e.g. view default database, view query output column names) and store them as * properties in metastore, if we need to create a permanent view. * + * Note: this is the v1 (session catalog) path. Permanent-view checks (no temp-object refs, + * no auto-generated aliases, no cycles) run at exec time here because Dataset-built commands + * can be constructed with `isAnalyzed=true` and bypass the analyzer's recapture path. The v2 + * equivalent is [[org.apache.spark.sql.catalyst.plans.logical.CreateView]]; its checks run at + * analysis time via [[CheckViewReferences]]. Mirror any new validation in both places. + * * @param name the name of this view. * @param userSpecifiedColumns the output column names and optional comments specified by users, * can be Nil if not specified. @@ -113,10 +119,10 @@ case class CreateViewCommand( if (userSpecifiedColumns.nonEmpty) { if (userSpecifiedColumns.length > analyzedPlan.output.length) { throw QueryCompilationErrors.cannotCreateViewNotEnoughColumnsError( - name, userSpecifiedColumns.map(_._1), analyzedPlan) + name.nameParts, userSpecifiedColumns.map(_._1), analyzedPlan) } else if (userSpecifiedColumns.length < analyzedPlan.output.length) { throw QueryCompilationErrors.cannotCreateViewTooManyColumnsError( - name, userSpecifiedColumns.map(_._1), analyzedPlan) + name.nameParts, userSpecifiedColumns.map(_._1), analyzedPlan) } if (viewSchemaMode == SchemaEvolution) { throw SparkException.internalError( @@ -128,8 +134,9 @@ case class CreateViewCommand( // When creating a permanent view, not allowed to reference temporary objects. // This should be called after `qe.assertAnalyzed()` (i.e., `child` can be resolved) - verifyTemporaryObjectsNotExists(isTemporary, name, analyzedPlan, referredTempFunctions) - verifyAutoGeneratedAliasesNotExists(analyzedPlan, isTemporary, name) + verifyTemporaryObjectsNotExists( + isTemporary, name.nameParts, analyzedPlan, referredTempFunctions) + verifyAutoGeneratedAliasesNotExists(analyzedPlan, isTemporary, name.nameParts) SchemaUtils.checkIndeterminateCollationInSchema(plan.schema) @@ -166,11 +173,13 @@ case class CreateViewCommand( // Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does nothing when the target view // already exists. } else if (tableMetadata.tableType != CatalogTableType.VIEW) { - throw QueryCompilationErrors.unsupportedCreateOrReplaceViewOnTableError(name, replace) + throw QueryCompilationErrors.unsupportedCreateOrReplaceViewOnTableError( + name.nameParts, replace) } else if (replace) { // Detect cyclic view reference on CREATE OR REPLACE VIEW. val viewIdent = tableMetadata.identifier - checkCyclicViewReference(analyzedPlan, Seq(viewIdent), viewIdent) + val viewFullIdent = tableMetadata.fullIdent + checkCyclicViewReference(analyzedPlan, Seq(viewFullIdent), viewFullIdent) // uncache the cached data before replacing an exists view logDebug(s"Try to uncache ${viewIdent.quotedString} before replacing.") @@ -186,7 +195,7 @@ case class CreateViewCommand( } else { // Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when the target view already // exists. - throw QueryCompilationErrors.viewAlreadyExistsError(name) + throw QueryCompilationErrors.viewAlreadyExistsError(name.nameParts) } } else { // Create the view if it doesn't exist. @@ -209,6 +218,12 @@ case class CreateViewCommand( * this command will try to alter a temporary view first, if view not exist, try permanent view * next, if still not exist, throw an exception. * + * Note: this is the v1 (session catalog) path. Permanent-view checks (no temp-object refs, + * no auto-generated aliases, no cycles) run at exec time here because Dataset-built commands + * can be constructed with `isAnalyzed=true` and bypass the analyzer's recapture path. The v2 + * equivalent is [[org.apache.spark.sql.catalyst.plans.logical.AlterViewAs]]; its checks run at + * analysis time via [[CheckViewReferences]]. Mirror any new validation in both places. + * * @param name the name of this view. * @param originalText the original SQL text of this view. Note that we can only alter a view by * SQL API, which means we always have originalText. @@ -242,8 +257,8 @@ case class AlterViewAsCommand( override def run(session: SparkSession): Seq[Row] = { val isTemporary = session.sessionState.catalog.isTempView(name) - verifyTemporaryObjectsNotExists(isTemporary, name, query, referredTempFunctions) - verifyAutoGeneratedAliasesNotExists(query, isTemporary, name) + verifyTemporaryObjectsNotExists(isTemporary, name.nameParts, query, referredTempFunctions) + verifyAutoGeneratedAliasesNotExists(query, isTemporary, name.nameParts) SchemaUtils.checkIndeterminateCollationInSchema(query.schema) if (isTemporary) { alterTemporaryView(session, query) @@ -277,7 +292,8 @@ case class AlterViewAsCommand( // Detect cyclic view reference on ALTER VIEW. val viewIdent = viewMeta.identifier - checkCyclicViewReference(analyzedPlan, Seq(viewIdent), viewIdent) + val viewFullIdent = viewMeta.fullIdent + checkCyclicViewReference(analyzedPlan, Seq(viewFullIdent), viewFullIdent) logDebug(s"Try to uncache ${viewIdent.quotedString} before replacing.") CommandUtils.uncacheTableOrView(session, viewIdent) @@ -559,16 +575,16 @@ object ViewHelper extends SQLConfHelper with Logging with CapturesConfig { * * @param plan the logical plan we detect cyclic view references from. * @param path the path between the altered view and current node. - * @param viewIdent the table identifier of the altered view, we compare two views by the - * `desc.identifier`. + * @param viewIdent the full multi-part identifier of the altered view. We compare two views by + * `desc.fullIdent` so multi-level namespaces (v2 catalogs) are distinguished. */ def checkCyclicViewReference( plan: LogicalPlan, - path: Seq[TableIdentifier], - viewIdent: TableIdentifier): Unit = { + path: Seq[Seq[String]], + viewIdent: Seq[String]): Unit = { plan match { case v: View => - val ident = v.desc.identifier + val ident = v.desc.fullIdent val newPath = path :+ ident // If the table identifier equals to the `viewIdent`, current view node is the same with // the altered view. We detect a view reference cycle, should throw an AnalysisException. @@ -594,12 +610,13 @@ object ViewHelper extends SQLConfHelper with Logging with CapturesConfig { } def verifyAutoGeneratedAliasesNotExists( - child: LogicalPlan, isTemporary: Boolean, name: TableIdentifier): Unit = { + child: LogicalPlan, isTemporary: Boolean, viewNameParts: Seq[String]): Unit = { if (!isTemporary && !conf.allowAutoGeneratedAliasForView) { child.output.foreach { attr => if (attr.metadata.contains("__autoGeneratedAlias")) { throw QueryCompilationErrors - .notAllowedToCreatePermanentViewWithoutAssigningAliasForExpressionError(name, attr) + .notAllowedToCreatePermanentViewWithoutAssigningAliasForExpressionError( + viewNameParts, attr) } } } @@ -610,7 +627,7 @@ object ViewHelper extends SQLConfHelper with Logging with CapturesConfig { */ def verifyTemporaryObjectsNotExists( isTemporary: Boolean, - name: TableIdentifier, + viewNameParts: Seq[String], child: LogicalPlan, referredTempFunctions: Seq[String]): Unit = { import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ @@ -618,16 +635,16 @@ object ViewHelper extends SQLConfHelper with Logging with CapturesConfig { val tempViews = collectTemporaryViews(child) tempViews.foreach { nameParts => throw QueryCompilationErrors.notAllowedToCreatePermanentViewByReferencingTempViewError( - name, nameParts.quoted) + viewNameParts, nameParts.quoted) } referredTempFunctions.foreach { funcName => throw QueryCompilationErrors.notAllowedToCreatePermanentViewByReferencingTempFuncError( - name, funcName) + viewNameParts, funcName) } val tempVars = collectTemporaryVariables(child) tempVars.foreach { nameParts => throw QueryCompilationErrors.notAllowedToCreatePermanentViewByReferencingTempVarError( - name.nameParts, nameParts) + viewNameParts, nameParts) } } } @@ -704,7 +721,7 @@ object ViewHelper extends SQLConfHelper with Logging with CapturesConfig { if (!storeAnalyzedPlanForView) { // Skip cyclic check because when stored analyzed plan for view, the depended // view is already converted to the underlying tables. So no cyclic views. - checkCyclicViewReference(analyzedPlan, Seq(name), name) + checkCyclicViewReference(analyzedPlan, Seq(name.nameParts), name.nameParts) } CommandUtils.uncacheTableOrView(session, name) } @@ -882,3 +899,71 @@ object ViewHelper extends SQLConfHelper with Logging with CapturesConfig { } } } + +/** + * Post-analysis check for v2 CREATE VIEW / ALTER VIEW. First rejects catalogs that do not + * implement [[ViewCatalog]] with `MISSING_CATALOG_ABILITY.VIEWS` -- we do this before the + * temp-object and auto-alias checks so a catalog that cannot host views at all surfaces the + * correct root cause instead of a misleading "references temp" error. Then rejects permanent + * views that reference temporary objects and view bodies with auto-generated aliases. + * `referredTempFunctions` is captured by the command's `markAsAnalyzed` before this rule runs. + * The v1 counterparts [[CreateViewCommand]] and [[AlterViewAsCommand]] keep their existing + * exec-time checks -- Dataset-built commands bypass the analyzer's re-capture path, so the + * exec-time safety net must stay for v1. + */ +object CheckViewReferences extends (LogicalPlan => Unit) { + import ViewHelper._ + + // Extract (catalog, identifier) for the two resolved shapes view commands reach us with: + // `ResolvedIdentifier` for CREATE VIEW, `ResolvedPersistentView` for ALTER VIEW. Other shapes + // are an analyzer bug. + private def catalogAndIdent(resolved: LogicalPlan): (CatalogPlugin, Identifier) = + resolved match { + case ri: ResolvedIdentifier => (ri.catalog, ri.identifier) + case rpv: ResolvedPersistentView => (rpv.catalog, rpv.identifier) + case other => + throw SparkException.internalError( + s"Unexpected child of view command: ${other.getClass.getName}") + } + + private def fullIdentFor(resolved: LogicalPlan): Seq[String] = { + val (catalog, ident) = catalogAndIdent(resolved) + catalog.name() +: ident.asMultipartIdentifier + } + + // Fail fast if the catalog cannot host views. Gate non-ViewCatalog plugins here so callers + // get the VIEWS-specific error rather than a generic cast failure later. + private def requireViewCatalog(resolved: LogicalPlan): Unit = { + val (catalog, _) = catalogAndIdent(resolved) + if (!catalog.isInstanceOf[ViewCatalog]) { + throw QueryCompilationErrors.missingCatalogViewsAbilityError(catalog) + } + } + + override def apply(plan: LogicalPlan): Unit = plan.foreach { + case cv: CreateView if cv.isAnalyzed => + requireViewCatalog(cv.child) + val fullIdent = fullIdentFor(cv.child) + verifyTemporaryObjectsNotExists( + isTemporary = false, fullIdent, cv.query, cv.referredTempFunctions) + verifyAutoGeneratedAliasesNotExists(cv.query, isTemporary = false, fullIdent) + // Cycles can only form when REPLACE'ing an existing view; a plain CREATE against an + // existing view fails earlier with `viewAlreadyExistsError` and against a non-existent + // view has nothing to cycle with. + if (cv.replace) { + checkCyclicViewReference(cv.query, Seq(fullIdent), fullIdent) + } + + case av: AlterViewAs if av.isAnalyzed => + // No capability check here: `Analyzer.lookupTableOrView(identifier, viewOnly=true)` + // already rejects non-ViewCatalog catalogs upstream for `UnresolvedView`, so by the time + // an AlterViewAs reaches this rule the catalog is guaranteed to be a ViewCatalog. + val fullIdent = fullIdentFor(av.child) + verifyTemporaryObjectsNotExists( + isTemporary = false, fullIdent, av.query, av.referredTempFunctions) + verifyAutoGeneratedAliasesNotExists(av.query, isTemporary = false, fullIdent) + checkCyclicViewReference(av.query, Seq(fullIdent), fullIdent) + + case _ => + } +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterV2ViewExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterV2ViewExec.scala new file mode 100644 index 0000000000000..cb21e773d86c4 --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterV2ViewExec.scala @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.datasources.v2 + +import scala.jdk.CollectionConverters._ + +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.analysis.{NoSuchViewException, ResolvedIdentifier, ViewSchemaMode} +import org.apache.spark.sql.catalyst.catalog.CatalogTable +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan +import org.apache.spark.sql.connector.catalog.{Identifier, TableCatalog, ViewCatalog, ViewInfo} +import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper +import org.apache.spark.sql.errors.QueryCompilationErrors +import org.apache.spark.sql.execution.command.CommandUtils + +/** + * Shared bits for the v2 ALTER VIEW ... AS exec. Loads the existing view once via + * `existingView` and uses it to preserve user-set TBLPROPERTIES, comment, collation, owner, + * and schema binding mode when constructing the replacement [[ViewInfo]]. A racing DDL between + * analysis and exec can change the target out from under us (dropped, or replaced with a + * non-view table); in that case we surface a regular no-such-view / not-a-view analysis error + * rather than propagating a stale analyzer decision. + * + * Transient fields (SQL configs, query column names) are re-captured from the + * current session by [[V2ViewPreparation.buildViewInfo]], matching v1 + * `AlterViewAsCommand.alterPermanentView`. PROP_OWNER and user TBLPROPERTIES flow through + * unchanged. + */ +private[v2] trait V2AlterViewPreparation extends V2ViewPreparation { + protected lazy val existingView: ViewInfo = try { + catalog.loadView(identifier) + } catch { + case _: NoSuchViewException => + // Race: the view disappeared after analysis. Surface no-such-view, or + // expect-view-not-table if a colliding non-view table appeared in a mixed catalog. + catalog match { + case tc: TableCatalog if tc.tableExists(identifier) => + throw QueryCompilationErrors.expectViewNotTableError( + (catalog.name() +: identifier.asMultipartIdentifier).toSeq, + cmd = "ALTER VIEW ... AS", + suggestAlternative = false, + t = this) + case _ => + throw new NoSuchViewException(identifier) + } + } + + protected lazy val existingProps: Map[String, String] = + existingView.properties.asScala.toMap + + private def existingProp(key: String): Option[String] = existingProps.get(key) + + // ALTER VIEW ... AS does not accept a user column list. + override def userSpecifiedColumns: Seq[(String, Option[String])] = Seq.empty + override def comment: Option[String] = existingProp(TableCatalog.PROP_COMMENT) + override def collation: Option[String] = existingProp(TableCatalog.PROP_COLLATION) + // Preserve the existing view's owner (v1-parity with AlterViewAsCommand's viewMeta.copy, + // which leaves `owner` untouched). If the existing view has no PROP_OWNER, pass it through + // as None so the replacement ViewInfo also has no owner. + override def owner: Option[String] = existingProp(TableCatalog.PROP_OWNER) + override def userProperties: Map[String, String] = existingProps + + // Preserve the existing view's schema binding mode. Reuse `viewSchemaModeFromProperties` + // for a v1-identical decode -- it honors `viewSchemaBindingEnabled` and defaults missing + // values to SchemaBinding. We feed the typed `ViewInfo.schemaMode` String in via a + // single-key map so the decode logic stays in one place. + override def viewSchemaMode: ViewSchemaMode = + CatalogTable.viewSchemaModeFromProperties( + Option(existingView.schemaMode) + .map(CatalogTable.VIEW_SCHEMA_MODE -> _) + .toMap) + + /** + * Force-evaluate `existingView` so `NoSuchViewException` / `expectViewNotTableError` + * surfaces before any other work (e.g. `buildViewInfo`, uncache, replace). The result is + * intentionally discarded; call this purely for its side effect of materializing the + * lazy val. + */ + protected def requireExistingView(): Unit = existingView +} + +/** + * Physical plan node for ALTER VIEW ... AS on a v2 [[ViewCatalog]]. Dispatches to + * [[ViewCatalog#replaceView]], which is contractually atomic. + */ +case class AlterV2ViewExec( + catalog: ViewCatalog, + identifier: Identifier, + originalText: String, + query: LogicalPlan) extends V2AlterViewPreparation { + + override protected def run(): Seq[InternalRow] = { + requireExistingView() + val info = buildViewInfo() + // Cyclic reference detection is done at analysis time in CheckViewReferences. + CommandUtils.uncacheTableOrView(session, ResolvedIdentifier(catalog, identifier)) + catalog.replaceView(identifier, info) + Seq.empty + } +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateV2ViewExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateV2ViewExec.scala new file mode 100644 index 0000000000000..6cfa95a2eaf43 --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateV2ViewExec.scala @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.datasources.v2 + +import scala.jdk.CollectionConverters._ + +import org.apache.spark.SparkException +import org.apache.spark.sql.catalyst.{CurrentUserContext, InternalRow} +import org.apache.spark.sql.catalyst.analysis.{ResolvedIdentifier, SchemaEvolution, ViewAlreadyExistsException, ViewSchemaMode} +import org.apache.spark.sql.catalyst.expressions.Attribute +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan +import org.apache.spark.sql.catalyst.util.CharVarcharUtils +import org.apache.spark.sql.connector.catalog.{Identifier, TableCatalog, ViewCatalog, ViewInfo} +import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper +import org.apache.spark.sql.errors.QueryCompilationErrors +import org.apache.spark.sql.execution.command.{CommandUtils, ViewHelper} +import org.apache.spark.sql.util.SchemaUtils +import org.apache.spark.util.ArrayImplicits._ + +/** + * Shared validation + ViewInfo construction for v2 CREATE VIEW / ALTER VIEW execs. + * + * Mirrors the persistent-view portion of v1 [[ViewHelper.prepareTable]] + the execution-time + * checks in [[org.apache.spark.sql.execution.command.CreateViewCommand.run]]. Post-analysis + * checks for temp-object references and auto-generated aliases run once for both v1 and v2 in + * [[org.apache.spark.sql.execution.command.CheckViewReferences]]. + */ +private[v2] trait V2ViewPreparation extends LeafV2CommandExec { + def catalog: ViewCatalog + def identifier: Identifier + def userSpecifiedColumns: Seq[(String, Option[String])] + def comment: Option[String] + def collation: Option[String] + def owner: Option[String] + def userProperties: Map[String, String] + def originalText: String + def query: LogicalPlan + def viewSchemaMode: ViewSchemaMode + + // Full multi-part identifier used for error rendering. Built once so we can avoid routing + // through the lossy v1 `TableIdentifier` for multi-level-namespace v2 catalogs. + protected lazy val fullNameParts: Seq[String] = + (catalog.name() +: identifier.asMultipartIdentifier).toSeq + + override def output: Seq[Attribute] = Seq.empty + + protected def buildViewInfo(): ViewInfo = { + import ViewHelper._ + + if (userSpecifiedColumns.nonEmpty) { + if (userSpecifiedColumns.length > query.output.length) { + throw QueryCompilationErrors.cannotCreateViewNotEnoughColumnsError( + fullNameParts, userSpecifiedColumns.map(_._1), query) + } else if (userSpecifiedColumns.length < query.output.length) { + throw QueryCompilationErrors.cannotCreateViewTooManyColumnsError( + fullNameParts, userSpecifiedColumns.map(_._1), query) + } + if (viewSchemaMode == SchemaEvolution) { + throw SparkException.internalError( + "View with user column list has viewSchemaMode EVOLUTION") + } + } + + SchemaUtils.checkIndeterminateCollationInSchema(query.schema) + + val aliasedSchema = CharVarcharUtils.getRawSchema( + aliasPlan(session, query, userSpecifiedColumns).schema, session.sessionState.conf) + SchemaUtils.checkColumnNameDuplication( + aliasedSchema.fieldNames.toImmutableArraySeq, session.sessionState.conf.resolver) + + val manager = session.sessionState.catalogManager + val queryColumnNames = if (viewSchemaMode == SchemaEvolution) { + Array.empty[String] + } else { + query.output.map(_.name).toArray + } + + val builder = new ViewInfo.Builder() + .withSchema(aliasedSchema) + .withProperties(userProperties.asJava) + .withQueryText(originalText) + .withCurrentCatalog(manager.currentCatalog.name) + .withCurrentNamespace(manager.currentNamespace) + .withSqlConfigs(sqlConfigsToProps(session.sessionState.conf, "").asJava) + .withSchemaMode(viewSchemaMode.toString) + .withQueryColumnNames(queryColumnNames) + // CREATE stamps the current user into PROP_OWNER (matching v2 CREATE TABLE via + // CatalogV2Util.withDefaultOwnership and v1 CREATE VIEW via CatalogTable.owner's default); + // ALTER preserves the existing view's owner (v1-parity with AlterViewAsCommand's + // viewMeta.copy). Both cases are expressed via the `owner` hook provided by the subclass. + owner.foreach(builder.withOwner) + comment.foreach(builder.withComment) + collation.foreach(builder.withCollation) + builder.build() + } + + protected def viewAlreadyExists(): Throwable = + QueryCompilationErrors.viewAlreadyExistsError(fullNameParts) +} + +/** + * Physical plan node for CREATE VIEW on a v2 [[ViewCatalog]]. Dispatches to + * [[ViewCatalog#createView]] for plain CREATE, [[ViewCatalog#createOrReplaceView]] for + * `OR REPLACE`, and short-circuits `IF NOT EXISTS` early via [[ViewCatalog#viewExists]] so + * the view body isn't analyzed when the view already exists. + */ +case class CreateV2ViewExec( + catalog: ViewCatalog, + identifier: Identifier, + userSpecifiedColumns: Seq[(String, Option[String])], + comment: Option[String], + collation: Option[String], + userProperties: Map[String, String], + originalText: String, + query: LogicalPlan, + allowExisting: Boolean, + replace: Boolean, + viewSchemaMode: ViewSchemaMode) extends V2ViewPreparation { + + override def owner: Option[String] = Some(CurrentUserContext.getCurrentUser) + + override protected def run(): Seq[InternalRow] = { + // CREATE VIEW IF NOT EXISTS: short-circuit before `buildViewInfo` if a view already sits + // at the ident -- avoids `aliasPlan` / config capture for the common no-op case (matches + // v1 `CreateViewCommand.run`). The mixed-catalog "table at ident" no-op is handled in the + // catch block below; that case is rare enough that paying for `buildViewInfo` is fine. + if (allowExisting && catalog.viewExists(identifier)) return Seq.empty + + val info = buildViewInfo() + try { + if (replace) { + CommandUtils.uncacheTableOrView(session, ResolvedIdentifier(catalog, identifier)) + catalog.createOrReplaceView(identifier, info) + } else { + catalog.createView(identifier, info) + } + } catch { + case _: ViewAlreadyExistsException => + // Catalog refused: something already occupies the ident. Decode whether it's a table + // (cross-type collision) or a view (race for plain CREATE / OR REPLACE), and emit the + // precise error -- or no-op for IF NOT EXISTS. + val isTable = catalog match { + case tc: TableCatalog => tc.tableExists(identifier) + case _ => false + } + if (isTable) { + if (!allowExisting) { + throw QueryCompilationErrors.unsupportedCreateOrReplaceViewOnTableError( + fullNameParts, replace) + } + // CREATE VIEW IF NOT EXISTS over a table is a no-op (v1 parity). + } else if (!allowExisting) { + throw viewAlreadyExists() + } + // else: a view appeared between our viewExists probe and createView; IF NOT EXISTS + // semantics make this a no-op. + } + Seq.empty + } +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala index 6730673cab025..d677ff1c4be2b 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala @@ -34,7 +34,7 @@ import org.apache.spark.sql.catalyst.plans.logical._ import org.apache.spark.sql.catalyst.trees.TreePattern.SCALAR_SUBQUERY import org.apache.spark.sql.catalyst.util.{toPrettySQL, GeneratedColumn, IdentityColumn, ResolveDefaultColumns, ResolveTableConstraints, V2ExpressionBuilder} import org.apache.spark.sql.classic.SparkSession -import org.apache.spark.sql.connector.catalog.{Identifier, StagingTableCatalog, SupportsDeleteV2, SupportsNamespaces, SupportsPartitionManagement, SupportsWrite, TableCapability, TableCatalog, TruncatableTable, V1Table} +import org.apache.spark.sql.connector.catalog.{Identifier, StagingTableCatalog, SupportsDeleteV2, SupportsNamespaces, SupportsPartitionManagement, SupportsWrite, TableCapability, TableCatalog, TruncatableTable, V1Table, ViewCatalog} import org.apache.spark.sql.connector.catalog.TableChange import org.apache.spark.sql.connector.catalog.index.SupportsIndex import org.apache.spark.sql.connector.expressions.{FieldReference, LiteralValue} @@ -301,6 +301,91 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat qualifyLocInTableSpec(tableSpec), orCreate = orCreate, invalidateCache) :: Nil } + // CheckViewReferences guarantees the catalog is a ViewCatalog by the time these strategy + // cases fire (it throws MISSING_CATALOG_ABILITY.VIEWS otherwise). + case CreateView(ResolvedIdentifier(catalog, ident), userSpecifiedColumns, comment, + collation, properties, originalText, child, allowExisting, replace, viewSchemaMode, + _, _) => + val sqlText = originalText.getOrElse { + throw QueryCompilationErrors.createPersistedViewFromDatasetAPINotAllowedError() + } + CreateV2ViewExec(catalog.asInstanceOf[ViewCatalog], ident, userSpecifiedColumns, comment, + collation, properties, sqlText, child, allowExisting, replace, viewSchemaMode) :: Nil + + case AlterViewAs(ResolvedPersistentView(catalog, ident, _), originalText, query, _, _) => + AlterV2ViewExec(catalog.asInstanceOf[ViewCatalog], ident, originalText, query) :: Nil + + // View DDL / inspection on a non-session v2 catalog that the v1 rewrite in + // `ResolveSessionCatalog` can't handle. These are tracked as follow-up work in SPARK-52729; + // pin the current failure mode with a clean `UNSUPPORTED_FEATURE.TABLE_OPERATION` error + // so users get a meaningful message (and test coverage catches a future regression to a + // generic planner error). + case SetViewProperties(ResolvedPersistentView(catalog, ident, _), _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "ALTER VIEW ... SET TBLPROPERTIES") + + case UnsetViewProperties(ResolvedPersistentView(catalog, ident, _), _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "ALTER VIEW ... UNSET TBLPROPERTIES") + + case AlterViewSchemaBinding(ResolvedPersistentView(catalog, ident, _), _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "ALTER VIEW ... WITH SCHEMA") + + case RenameTable(ResolvedPersistentView(catalog, ident, _), _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "ALTER VIEW ... RENAME TO") + + case ShowCreateTable(ResolvedPersistentView(catalog, ident, _), _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "SHOW CREATE TABLE") + + case ShowTableProperties(ResolvedPersistentView(catalog, ident, _), _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "SHOW TBLPROPERTIES") + + case ShowColumns(ResolvedPersistentView(catalog, ident, _), _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "SHOW COLUMNS") + + case DescribeRelation(ResolvedPersistentView(catalog, ident, _), _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "DESCRIBE TABLE") + + case DescribeColumn(ResolvedPersistentView(catalog, ident, _), _, _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "DESCRIBE TABLE ... COLUMN") + + // Plans that resolve through `UnresolvedTableOrView` reach here with a + // `ResolvedPersistentView` child for non-session v2 views (the v1 rewrite in + // `ResolveSessionCatalog` no longer matches them because `ResolvedViewIdentifier` is gated + // on `isSessionCatalog`). Pin each with `UNSUPPORTED_FEATURE.TABLE_OPERATION` so users get + // a clean `AnalysisException` instead of a generic "No plan for ..." assertion from the + // planner. Tracked for follow-up real handlers in SPARK-52729. + case RefreshTable(ResolvedPersistentView(catalog, ident, _)) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "REFRESH TABLE") + + case AnalyzeTable(ResolvedPersistentView(catalog, ident, _), _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "ANALYZE TABLE") + + case AnalyzeColumn(ResolvedPersistentView(catalog, ident, _), _, _) => + throw QueryCompilationErrors.unsupportedTableOperationError( + catalog, ident, "ANALYZE TABLE ... FOR COLUMNS") + + // SHOW PARTITIONS on a view is already rejected during analysis: the parser uses + // `UnresolvedTable` (not `UnresolvedTableOrView`), so `CheckAnalysis` surfaces + // `EXPECT_TABLE_NOT_VIEW.NO_ALTERNATIVE` before planning. No strategy case needed. + + // DROP VIEW on a non-session ViewCatalog. The v1 rewrite in `ResolveSessionCatalog` skips + // ViewCatalog catalogs, so they fall through here. `DropViewExec` calls + // `ViewCatalog.dropView` and surfaces `EXPECT_VIEW_NOT_TABLE` if the identifier resolves to + // a table in a mixed catalog. + case DropView(r @ ResolvedIdentifier(catalog: ViewCatalog, ident), ifExists) => + val invalidateFunc = () => CommandUtils.uncacheTableOrView(session, r) + DropViewExec(catalog, ident, ifExists, invalidateFunc) :: Nil + case ReplaceTableAsSelect(ResolvedIdentifier(catalog, ident), parts, query, tableSpec: TableSpec, options, orCreate, true) => catalog match { @@ -493,6 +578,15 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat case ShowTables(ResolvedNamespace(catalog, ns, _), pattern, output) => ShowTablesExec(output, catalog.asTableCatalog, ns, pattern) :: Nil + // SHOW VIEWS on a v2 ViewCatalog. `ResolveSessionCatalog` rewrites the SHOW VIEWS plan to + // v1 `ShowViewsCommand` only when the catalog is NOT a `ViewCatalog`; non-`ViewCatalog` + // catalogs (session or not) are rejected with `MISSING_CATALOG_ABILITY.VIEWS` there. So + // this case sees `ViewCatalog` catalogs (typically non-session, since the default + // `V2SessionCatalog` is not a `ViewCatalog`; a session-catalog override that mixes in + // `ViewCatalog` would also reach here). + case ShowViews(ResolvedNamespace(catalog: ViewCatalog, ns, _), pattern, output) => + ShowViewsExec(output, catalog, ns, pattern) :: Nil + case ShowTablesExtended( ResolvedNamespace(catalog, ns, _), pattern, diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTableExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTableExec.scala index c94af4e3dceb3..18e6a5eb86ac8 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTableExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTableExec.scala @@ -19,12 +19,22 @@ package org.apache.spark.sql.execution.datasources.v2 import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions.Attribute -import org.apache.spark.sql.connector.catalog.{Identifier, TableCatalog} +import org.apache.spark.sql.connector.catalog.{Identifier, TableCatalog, ViewCatalog} import org.apache.spark.sql.errors.QueryCompilationErrors import org.apache.spark.util.ArrayImplicits._ /** * Physical plan node for dropping a table. + * + * Probes `tableExists` upfront so `IF EXISTS` over a missing table is a clean no-op even + * on catalogs whose `dropTable` / `purgeTable` does not honor the "return false on missing" + * contract (e.g. JDBC catalogs that throw a SQL syntax error, or the default `purgeTable` + * that throws `UNSUPPORTED_FEATURE.PURGE_TABLE` unconditionally). + * + * When the table is absent, falls back to `viewExists` for catalogs that also implement + * [[ViewCatalog]] -- distinguishes "wrong type" from "missing" so a `DROP TABLE someView` + * on a mixed catalog surfaces the dedicated `EXPECT_TABLE_NOT_VIEW` error rather than a + * generic "table not found", matching the v1 `DropTableCommand(isView = false)` behavior. */ case class DropTableExec( catalog: TableCatalog, @@ -37,9 +47,18 @@ case class DropTableExec( if (catalog.tableExists(ident)) { invalidateCache() if (purge) catalog.purgeTable(ident) else catalog.dropTable(ident) - } else if (!ifExists) { - val nameParts = (catalog.name() +: ident.namespace() :+ ident.name()).toImmutableArraySeq - throw QueryCompilationErrors.noSuchTableError(nameParts) + } else { + val nameParts = + (catalog.name() +: ident.namespace() :+ ident.name()).toImmutableArraySeq + catalog match { + case vc: ViewCatalog if vc.viewExists(ident) => + throw QueryCompilationErrors.expectTableNotViewError( + nameParts, cmd = "DROP TABLE", suggestAlternative = false, t = this) + case _ if !ifExists => + throw QueryCompilationErrors.noSuchTableError(nameParts) + case _ => + // IF EXISTS: no-op. + } } Seq.empty diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropViewExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropViewExec.scala new file mode 100644 index 0000000000000..9a665f644e0de --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropViewExec.scala @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.datasources.v2 + +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.analysis.NoSuchViewException +import org.apache.spark.sql.catalyst.expressions.Attribute +import org.apache.spark.sql.connector.catalog.{Identifier, TableCatalog, ViewCatalog} +import org.apache.spark.sql.errors.QueryCompilationErrors +import org.apache.spark.util.ArrayImplicits._ + +/** + * Physical plan node for DROP VIEW on a v2 [[ViewCatalog]]. Calls [[ViewCatalog#dropView]]; if + * it returns false and the catalog also implements [[TableCatalog]] with a table at this + * identifier, surfaces the dedicated `EXPECT_VIEW_NOT_TABLE` error rather than a generic + * "view not found" -- matching v1 `DropTableCommand(isView = true)`. + */ +case class DropViewExec( + catalog: ViewCatalog, + ident: Identifier, + ifExists: Boolean, + invalidateCache: () => Unit) extends LeafV2CommandExec { + + override protected def run(): Seq[InternalRow] = { + val dropped = catalog.dropView(ident) + if (dropped) { + invalidateCache() + } else { + val nameParts = + (catalog.name() +: ident.namespace() :+ ident.name()).toImmutableArraySeq + catalog match { + case tc: TableCatalog if tc.tableExists(ident) => + throw QueryCompilationErrors.expectViewNotTableError( + nameParts, cmd = "DROP VIEW", suggestAlternative = false, t = this) + case _ if !ifExists => + throw new NoSuchViewException(ident) + case _ => + // IF EXISTS: no-op. + } + } + Seq.empty + } + + override def output: Seq[Attribute] = Seq.empty +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowViewsExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowViewsExec.scala new file mode 100644 index 0000000000000..00927f05842ad --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowViewsExec.scala @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.datasources.v2 + +import scala.collection.mutable.ArrayBuffer + +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.expressions.Attribute +import org.apache.spark.sql.catalyst.util.StringUtils +import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.NamespaceHelper +import org.apache.spark.sql.connector.catalog.ViewCatalog +import org.apache.spark.sql.execution.LeafExecNode + +/** + * Physical plan node for SHOW VIEWS on a v2 [[ViewCatalog]]. Enumerates view identifiers via + * [[ViewCatalog#listViews]]. v2 catalogs have no temp views, so the {@code isTemporary} column + * is always false -- mirroring v1 {@code ShowViewsCommand}, which sets {@code isTemporary=true} + * only for local/global temp views that live in the session catalog. + */ +case class ShowViewsExec( + output: Seq[Attribute], + catalog: ViewCatalog, + namespace: Seq[String], + pattern: Option[String]) extends V2CommandExec with LeafExecNode { + override protected def run(): Seq[InternalRow] = { + val rows = new ArrayBuffer[InternalRow]() + catalog.listViews(namespace.toArray).foreach { ident => + val nameMatches = + pattern.forall(p => StringUtils.filterPattern(Seq(ident.name), p).nonEmpty) + if (nameMatches) { + rows += toCatalystRow(ident.namespace().quoted, ident.name(), false) + } + } + rows.toSeq + } +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala b/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala index 9bd68cbe72a07..d8fe14a0664c1 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala @@ -35,7 +35,7 @@ import org.apache.spark.sql.execution.{ColumnarRule, CommandExecutionMode, Query import org.apache.spark.sql.execution.adaptive.AdaptiveRulesHolder import org.apache.spark.sql.execution.aggregate.{ResolveEncodersInScalaAgg, ScalaUDAF} import org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin -import org.apache.spark.sql.execution.command.CommandCheck +import org.apache.spark.sql.execution.command.{CheckViewReferences, CommandCheck} import org.apache.spark.sql.execution.datasources._ import org.apache.spark.sql.execution.datasources.v2.{TableCapabilityCheck, V2SessionCatalog} import org.apache.spark.sql.execution.streaming.runtime.ResolveWriteToStream @@ -259,6 +259,7 @@ abstract class BaseSessionStateBuilder( HiveOnlyCheck +: TableCapabilityCheck +: CommandCheck +: + CheckViewReferences +: ViewSyncSchemaToMetaStore +: customCheckRules } diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/explain-aqe.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/explain-aqe.sql.out index 4b9bb859cd567..3f16d4f756511 100644 --- a/sql/core/src/test/resources/sql-tests/analyzer-results/explain-aqe.sql.out +++ b/sql/core/src/test/resources/sql-tests/analyzer-results/explain-aqe.sql.out @@ -174,7 +174,7 @@ EXPLAIN FORMATTED CREATE VIEW explain_view AS SELECT key, val FROM explain_temp1 -- !query analysis -ExplainCommand 'CreateView SELECT key, val FROM explain_temp1, false, false, COMPENSATION, FormattedMode +ExplainCommand 'CreateView SELECT key, val FROM explain_temp1, false, false, COMPENSATION, false, FormattedMode -- !query diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/explain.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/explain.sql.out index 4b9bb859cd567..3f16d4f756511 100644 --- a/sql/core/src/test/resources/sql-tests/analyzer-results/explain.sql.out +++ b/sql/core/src/test/resources/sql-tests/analyzer-results/explain.sql.out @@ -174,7 +174,7 @@ EXPLAIN FORMATTED CREATE VIEW explain_view AS SELECT key, val FROM explain_temp1 -- !query analysis -ExplainCommand 'CreateView SELECT key, val FROM explain_temp1, false, false, COMPENSATION, FormattedMode +ExplainCommand 'CreateView SELECT key, val FROM explain_temp1, false, false, COMPENSATION, false, FormattedMode -- !query diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataOnlyTableSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataOnlyTableSuite.scala new file mode 100644 index 0000000000000..8d3ad19419dff --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataOnlyTableSuite.scala @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector + +import org.apache.spark.SparkConf +import org.apache.spark.sql.{QueryTest, Row} +import org.apache.spark.sql.catalyst.analysis.NoSuchTableException +import org.apache.spark.sql.connector.catalog.{Identifier, MetadataOnlyTable, Table, TableCatalog, TableChange, TableInfo, TableSummary} +import org.apache.spark.sql.connector.expressions.LogicalExpressions +import org.apache.spark.sql.test.SharedSparkSession +import org.apache.spark.sql.types.StructType +import org.apache.spark.sql.util.CaseInsensitiveStringMap + +/** + * Tests for the data-source-table side of [[MetadataOnlyTable]]: a v2 catalog returns + * metadata-only tables and Spark reads / writes them via the V1 data-source path. + * View-related paths live in [[DataSourceV2MetadataOnlyViewSuite]]. + */ +class DataSourceV2MetadataOnlyTableSuite extends QueryTest with SharedSparkSession { + import testImplicits._ + + override def sparkConf: SparkConf = super.sparkConf + .set( + "spark.sql.catalog.table_catalog", + classOf[TestingDataSourceTableCatalog].getName) + + test("file source table") { + withTempPath { path => + val loc = path.getCanonicalPath + val tableName = s"table_catalog.`$loc`.test_json" + + spark.range(10).select($"id".cast("string").as("col")).write.json(loc) + checkAnswer(spark.table(tableName), 0.until(10).map(i => Row(i.toString))) + + sql(s"INSERT INTO $tableName SELECT 'abc'") + checkAnswer(spark.table(tableName), 0.until(10).map(i => Row(i.toString)) :+ Row("abc")) + + sql(s"INSERT OVERWRITE $tableName SELECT 'xyz'") + checkAnswer(spark.table(tableName), Row("xyz")) + } + } + + test("partitioned file source table") { + withTempPath { path => + val loc = path.getCanonicalPath + val tableName = s"table_catalog.`$loc`.test_partitioned_json" + + Seq(1 -> 1, 2 -> 1).toDF("c1", "c2").write.partitionBy("c2").json(loc) + checkAnswer(spark.table(tableName), Seq(Row(1, 1), Row(2, 1))) + + sql(s"INSERT INTO $tableName SELECT 1, 2") + checkAnswer(spark.table(tableName), Seq(Row(1, 1), Row(2, 1), Row(1, 2))) + + sql(s"INSERT INTO $tableName PARTITION(c2=3) SELECT 1") + checkAnswer(spark.table(tableName), Seq(Row(1, 1), Row(2, 1), Row(1, 2), Row(1, 3))) + + sql(s"INSERT OVERWRITE $tableName PARTITION(c2=2) SELECT 10") + checkAnswer(spark.table(tableName), Seq(Row(1, 1), Row(2, 1), Row(10, 2), Row(1, 3))) + + sql(s"INSERT OVERWRITE $tableName SELECT 20, 20") + checkAnswer(spark.table(tableName), Row(20, 20)) + } + } + + // TODO: move the v2 data source table handling from V2SessionCatalog to the analyzer + ignore("v2 data source table") { + val tableName = "table_catalog.default.test_v2" + checkAnswer(spark.table(tableName), 0.until(10).map(i => Row(i, -i))) + } + + test("DESCRIBE TABLE EXTENDED on a non-view MetadataOnlyTable shows the real identifier") { + // MetadataOnlyTable.name() is read by DescribeTableExec's "Name" row. Pin that it + // reflects the catalog-supplied identifier (here TestingDataSourceTableCatalog passes + // `ident.toString`) rather than a generic placeholder, so the DESCRIBE output is + // meaningful for users. + withTempPath { path => + val loc = path.getCanonicalPath + val tableName = s"table_catalog.`$loc`.test_json" + spark.range(1).select($"id".cast("string").as("col")).write.json(loc) + val nameRow = sql(s"DESCRIBE TABLE EXTENDED $tableName") + .collect() + .find(_.getString(0) == "Name") + .getOrElse(fail("DESCRIBE output missing the `Name` row")) + val rendered = nameRow.getString(1) + assert(rendered.contains("test_json"), s"expected the real identifier, got: $rendered") + } + } + + test("fully-qualified column reference uses the real catalog name") { + withTempPath { path => + val loc = path.getCanonicalPath + val tableName = s"table_catalog.`$loc`.test_json" + + spark.range(3).select($"id".cast("string").as("col")).write.json(loc) + + // 1-part and 2-part references resolve via last-part suffix matching. + checkAnswer( + sql(s"SELECT test_json.col FROM $tableName"), + Seq(Row("0"), Row("1"), Row("2"))) + checkAnswer( + sql(s"SELECT `$loc`.test_json.col FROM $tableName"), + Seq(Row("0"), Row("1"), Row("2"))) + + // 3-part reference uses the real catalog name. `V1Table.toCatalogTable` sets + // `CatalogTable.multipartIdentifier` to `[table_catalog, , test_json]`; the + // SessionCatalog change in this PR makes `getRelation` prefer that over the hardcoded + // `spark_catalog` qualifier, so the SubqueryAlias carries the real catalog and this + // 3-part column ref resolves. + checkAnswer( + sql(s"SELECT $tableName.col FROM $tableName"), + Seq(Row("0"), Row("1"), Row("2"))) + } + } +} + +/** + * A read-only [[TableCatalog]] that returns [[MetadataOnlyTable]] for a small set of canned + * table fixtures. Used to drive the data-source-table read path (file source + v2 provider) + * through Spark's V1 data-source machinery. + */ +class TestingDataSourceTableCatalog extends TableCatalog { + override def loadTable(ident: Identifier): Table = ident.name() match { + case "test_json" => + val info = new TableInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withProvider("json") + .withLocation(ident.namespace().head) + .withTableType(TableSummary.EXTERNAL_TABLE_TYPE) + .build() + new MetadataOnlyTable(info, ident.toString) + case "test_partitioned_json" => + val partitioning = LogicalExpressions.identity(LogicalExpressions.reference(Seq("c2"))) + val info = new TableInfo.Builder() + .withSchema(new StructType().add("c1", "int").add("c2", "int")) + .withProvider("json") + .withLocation(ident.namespace().head) + .withTableType(TableSummary.EXTERNAL_TABLE_TYPE) + .withPartitions(Array(partitioning)) + .build() + new MetadataOnlyTable(info, ident.toString) + case "test_v2" => + val info = new TableInfo.Builder() + .withSchema(FakeV2Provider.schema) + .withProvider(classOf[FakeV2Provider].getName) + .build() + new MetadataOnlyTable(info, ident.toString) + case _ => throw new NoSuchTableException(ident) + } + + override def createTable(ident: Identifier, info: TableInfo): Table = + throw new RuntimeException("shouldn't be called") + override def alterTable(ident: Identifier, changes: TableChange*): Table = + throw new RuntimeException("shouldn't be called") + override def dropTable(ident: Identifier): Boolean = + throw new RuntimeException("shouldn't be called") + override def renameTable(oldIdent: Identifier, newIdent: Identifier): Unit = + throw new RuntimeException("shouldn't be called") + override def listTables(namespace: Array[String]): Array[Identifier] = + throw new RuntimeException("shouldn't be called") + + private var catalogName = "" + override def initialize(name: String, options: CaseInsensitiveStringMap): Unit = { + catalogName = name + } + override def name(): String = catalogName +} diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataOnlyViewSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataOnlyViewSuite.scala new file mode 100644 index 0000000000000..0851e6d2df765 --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2MetadataOnlyViewSuite.scala @@ -0,0 +1,1120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector + +import org.apache.spark.SparkConf +import org.apache.spark.sql.{AnalysisException, QueryTest, Row} +import org.apache.spark.sql.catalyst.analysis.{NoSuchTableException, NoSuchViewException, TableAlreadyExistsException, ViewAlreadyExistsException} +import org.apache.spark.sql.connector.catalog.{Identifier, MetadataOnlyTable, RelationCatalog, Table, TableCatalog, TableChange, TableInfo, TableSummary, V1Table, ViewCatalog, ViewInfo} +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.test.SharedSparkSession +import org.apache.spark.sql.types.StructType +import org.apache.spark.sql.util.CaseInsensitiveStringMap + +/** + * Tests for the view side of [[MetadataOnlyTable]]: view-text expansion on read, and + * CREATE VIEW / ALTER VIEW ... AS going through the v2 write path + * (`CreateV2ViewExec` / `AlterV2ViewExec`). View writes route through + * [[ViewCatalog#createView]] / [[ViewCatalog#replaceView]]. + * Data-source-table read paths live in + * [[org.apache.spark.sql.connector.DataSourceV2MetadataOnlyTableSuite]]. + * + * TODO: once the remaining v2 view DDL is implemented (SET/UNSET TBLPROPERTIES, SHOW CREATE + * VIEW, RENAME TO, SCHEMA BINDING, DESCRIBE / SHOW TBLPROPERTIES on v2 views), register a + * `MetadataOnlyTable`-backed `DelegatingCatalogExtension` as `spark.sql.catalog.spark_catalog` + * and run the shared [[org.apache.spark.sql.execution.PersistedViewTestSuite]] body against + * the v2 path for full parity with the v1 persisted-view coverage. + */ +class DataSourceV2MetadataOnlyViewSuite extends QueryTest with SharedSparkSession { + import testImplicits._ + + override def sparkConf: SparkConf = super.sparkConf + .set("spark.sql.catalog.view_catalog", classOf[TestingRelationCatalog].getName) + + // --- View read path ----------------------------------------------------- + + test("read view expands SQL text and applies captured SQL configs") { + withTable("spark_catalog.default.t") { + Seq("a", "b").toDF("col").write.saveAsTable("spark_catalog.default.t") + // view_catalog.ansi.test_view stores view.sqlConfig.spark.sql.ansi.enabled=true; + // view_catalog.non_ansi.test_view stores it =false. The view body does + // `col::int` which errors in ANSI mode and yields NULL in non-ANSI mode. + intercept[Exception](spark.table("view_catalog.ansi.test_view").collect()) + checkAnswer(spark.table("view_catalog.non_ansi.test_view"), Row("b", null)) + } + } + + test("read view resolves unqualified refs via captured current catalog/namespace") { + withTable("spark_catalog.default.t") { + Seq("a", "b").toDF("col").write.saveAsTable("spark_catalog.default.t") + // View text uses the unqualified name `t`; it resolves via the stored + // current catalog / namespace properties. + checkAnswer(spark.table("view_catalog.ns.test_unqualified_view"), Row("b")) + } + } + + test("read view resolves unqualified refs via multi-part captured namespace") { + // End-to-end coverage of the v2 encoder -> parser round-trip: test_unqualified_multi is a + // view whose captured catalog+namespace is view_catalog.ns1.ns2 (two-part namespace) and + // whose body references `t` unqualified. At read time the unqualified `t` must expand to + // view_catalog.ns1.ns2.t via the captured context -- which TestingRelationCatalog resolves to + // its own `t` fixture at that namespace. + checkAnswer( + spark.table("view_catalog.outer_ns.test_unqualified_multi"), + Row("multi")) + } + + // --- ViewInfo unit tests ----------------------------------------------- + + test("multi-part captured namespace round-trips through V1Table.toCatalogTable") { + // (a) ViewInfo.Builder stores (cat, Array(db1, db2)) as typed fields. + // (b) V1Table.toCatalogTable reads them directly and emits v1's numbered + // view.catalogAndNamespace.* keys so (c) the resulting CatalogTable's + // `viewCatalogAndNamespace` exposes the full (cat, db1, db2), which is what the v1 + // view-resolution path consumes to expand unqualified references in the view body. + val info = new ViewInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withQueryText("SELECT col FROM t") + .withCurrentCatalog("my_cat") + .withCurrentNamespace(Array("db1", "db2")) + .build() + val motTable = new MetadataOnlyTable(info, "v") + // Any CatalogPlugin works here; toCatalogTable only reads `catalog.name()`. + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + val ct = V1Table.toCatalogTable( + catalog, Identifier.of(Array("ns"), "v"), motTable) + assert(ct.viewCatalogAndNamespace == Seq("my_cat", "db1", "db2")) + + // Namespace parts containing dots flow through structurally (no string encoding). + val infoWeird = new ViewInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withQueryText("SELECT col FROM t") + .withCurrentCatalog("my_cat") + .withCurrentNamespace(Array("weird.db", "normal")) + .build() + val ctWeird = V1Table.toCatalogTable( + catalog, Identifier.of(Array("ns"), "v"), new MetadataOnlyTable(infoWeird, "v")) + assert(ctWeird.viewCatalogAndNamespace == Seq("my_cat", "weird.db", "normal")) + } + + test("view with no captured catalog omits viewCatalogAndNamespace") { + val info = new ViewInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withQueryText("SELECT * FROM spark_catalog.default.t") + .build() + val motTable = new MetadataOnlyTable(info, "v") + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + val ct = V1Table.toCatalogTable(catalog, Identifier.of(Array("ns"), "v"), motTable) + assert(ct.viewCatalogAndNamespace.isEmpty) + } + + // --- CREATE VIEW on a plain TableCatalog -------------------------------- + + test("CREATE VIEW on a v2 catalog") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.my_view AS " + + "SELECT x FROM spark_catalog.default.t WHERE x > 1") + checkAnswer(spark.table("view_catalog.default.my_view"), Seq(Row(2), Row(3))) + } + } + + test("CREATE VIEW IF NOT EXISTS is a no-op when the view exists") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_ifne AS " + + "SELECT x FROM spark_catalog.default.t") + // Re-running with IF NOT EXISTS should not fail and should not change the view. + sql("CREATE VIEW IF NOT EXISTS view_catalog.default.v_ifne AS " + + "SELECT x + 100 AS x FROM spark_catalog.default.t") + checkAnswer(spark.table("view_catalog.default.v_ifne"), + Seq(Row(1), Row(2), Row(3))) + } + } + + test("CREATE VIEW without IF NOT EXISTS fails when the view exists") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_dup AS " + + "SELECT x FROM spark_catalog.default.t") + intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.default.v_dup AS " + + "SELECT x FROM spark_catalog.default.t") + } + } + } + + test("CREATE OR REPLACE VIEW replaces an existing view") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_replace AS " + + "SELECT x FROM spark_catalog.default.t WHERE x > 10") + checkAnswer(spark.table("view_catalog.default.v_replace"), Seq.empty[Row]) + sql("CREATE OR REPLACE VIEW view_catalog.default.v_replace AS " + + "SELECT x FROM spark_catalog.default.t WHERE x > 1") + checkAnswer(spark.table("view_catalog.default.v_replace"), Seq(Row(2), Row(3))) + } + } + + test("CREATE VIEW on a catalog without ViewCatalog fails") { + withSQLConf( + "spark.sql.catalog.no_view_catalog" -> classOf[TestingTableOnlyCatalog].getName) { + val ex = intercept[AnalysisException] { + sql("CREATE VIEW no_view_catalog.default.v AS SELECT 1") + } + assert(ex.getCondition == "MISSING_CATALOG_ABILITY.VIEWS") + } + } + + test("CREATE VIEW rejects too-few / too-many user-specified columns") { + withTable("spark_catalog.default.t") { + Seq(1 -> 10).toDF("x", "y").write.saveAsTable("spark_catalog.default.t") + intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.default.v_few (a) AS " + + "SELECT x, y FROM spark_catalog.default.t") + } + intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.default.v_many (a, b, c) AS " + + "SELECT x, y FROM spark_catalog.default.t") + } + } + } + + test("CREATE VIEW rejects reference to a temporary function") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + spark.udf.register("temp_udf", (i: Int) => i + 1) + val ex = intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.default.v_tempfn AS " + + "SELECT temp_udf(x) FROM spark_catalog.default.t") + } + assert(ex.getMessage.toLowerCase(java.util.Locale.ROOT).contains("temporary")) + } + } + + test("CREATE VIEW rejects reference to a temporary view") { + withTempView("tv") { + spark.range(3).createOrReplaceTempView("tv") + val ex = intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.default.v_tempview AS SELECT id FROM tv") + } + assert(ex.getMessage.toLowerCase(java.util.Locale.ROOT).contains("temporary")) + } + } + + test("CREATE VIEW rejects reference to a temporary variable") { + withSessionVariable("temp_var") { + sql("DECLARE VARIABLE temp_var INT DEFAULT 1") + val ex = intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.default.v_tempvar AS SELECT temp_var AS x") + } + assert(ex.getMessage.toLowerCase(java.util.Locale.ROOT).contains("temporary")) + } + } + + test("CREATE VIEW propagates DEFAULT COLLATION to TableInfo") { + withTable("spark_catalog.default.t") { + Seq("a", "b").toDF("col").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_coll DEFAULT COLLATION UTF8_BINARY AS " + + "SELECT col FROM spark_catalog.default.t") + // TestingRelationCatalog stores the TableInfo verbatim, so the collation property is + // observable via the catalog-stored builder output. + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + val info = catalog.getStoredView(Array("default"), "v_coll") + assert(info.properties().get(TableCatalog.PROP_COLLATION) == "UTF8_BINARY") + } + } + + test("CREATE OR REPLACE VIEW detects cyclic view references") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_cycle_a AS " + + "SELECT x FROM spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_cycle_b AS " + + "SELECT x FROM view_catalog.default.v_cycle_a") + val ex = intercept[AnalysisException] { + sql("CREATE OR REPLACE VIEW view_catalog.default.v_cycle_a AS " + + "SELECT x FROM view_catalog.default.v_cycle_b") + } + assert(ex.getCondition == "RECURSIVE_VIEW") + } + } + + test("CREATE VIEW over a non-view table entry is rejected (plain TableCatalog)") { + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + val tableIdent = Identifier.of(Array("default"), "v_existing_table") + val tableInfo = new TableInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withTableType(TableSummary.EXTERNAL_TABLE_TYPE) + .build() + catalog.createTable(tableIdent, tableInfo) + try { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + + // CREATE OR REPLACE VIEW must not silently destroy a non-view table -- v1 parity. + val replaceEx = intercept[AnalysisException] { + sql("CREATE OR REPLACE VIEW view_catalog.default.v_existing_table AS " + + "SELECT x FROM spark_catalog.default.t") + } + assert(replaceEx.getCondition == "EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE") + + // Plain CREATE VIEW over a table surfaces TABLE_OR_VIEW_ALREADY_EXISTS, matching v1. + val createEx = intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.default.v_existing_table AS " + + "SELECT x FROM spark_catalog.default.t") + } + assert(createEx.getCondition == "TABLE_OR_VIEW_ALREADY_EXISTS") + + // CREATE VIEW IF NOT EXISTS is a no-op -- the table entry is untouched. + sql("CREATE VIEW IF NOT EXISTS view_catalog.default.v_existing_table AS " + + "SELECT x FROM spark_catalog.default.t") + val stored = catalog.getStoredInfo(Array("default"), "v_existing_table") + assert(!stored.isInstanceOf[ViewInfo]) + assert(stored.properties().get(TableCatalog.PROP_TABLE_TYPE) == + TableSummary.EXTERNAL_TABLE_TYPE) + } + } finally { + catalog.dropTable(tableIdent) + } + } + + // --- ALTER VIEW --------------------------------------------------------- + + test("ALTER VIEW ... AS updates the view body on a v2 catalog") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_alter AS " + + "SELECT x FROM spark_catalog.default.t WHERE x > 10") + checkAnswer(spark.table("view_catalog.default.v_alter"), Seq.empty[Row]) + + sql("ALTER VIEW view_catalog.default.v_alter AS " + + "SELECT x FROM spark_catalog.default.t WHERE x > 1") + checkAnswer(spark.table("view_catalog.default.v_alter"), Seq(Row(2), Row(3))) + } + } + + test("ALTER VIEW on a missing view fails at analysis") { + // UnresolvedView resolves through lookupTableOrView and the missing view surfaces as an + // AnalysisException before we ever reach the v2 exec. The exact error condition (e.g. + // TABLE_OR_VIEW_NOT_FOUND) varies across Spark versions; we just assert we fail cleanly. + intercept[AnalysisException] { + sql("ALTER VIEW view_catalog.default.does_not_exist AS SELECT 1 AS x") + } + } + + test("ALTER VIEW rejects reference to a temporary function") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_alter_tempfn AS " + + "SELECT x FROM spark_catalog.default.t") + spark.udf.register("temp_udf_alter", (i: Int) => i + 1) + val ex = intercept[AnalysisException] { + sql("ALTER VIEW view_catalog.default.v_alter_tempfn AS " + + "SELECT temp_udf_alter(x) FROM spark_catalog.default.t") + } + assert(ex.getMessage.toLowerCase(java.util.Locale.ROOT).contains("temporary")) + } + } + + test("ALTER VIEW rejects reference to a temporary view") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_alter_tempview AS " + + "SELECT x FROM spark_catalog.default.t") + withTempView("tv_alter") { + spark.range(3).createOrReplaceTempView("tv_alter") + val ex = intercept[AnalysisException] { + sql("ALTER VIEW view_catalog.default.v_alter_tempview AS SELECT id FROM tv_alter") + } + assert(ex.getMessage.toLowerCase(java.util.Locale.ROOT).contains("temporary")) + } + } + } + + test("ALTER VIEW rejects reference to a temporary variable") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_alter_tempvar AS " + + "SELECT x FROM spark_catalog.default.t") + withSessionVariable("temp_var_alter") { + sql("DECLARE VARIABLE temp_var_alter INT DEFAULT 1") + val ex = intercept[AnalysisException] { + sql("ALTER VIEW view_catalog.default.v_alter_tempvar AS SELECT temp_var_alter AS x") + } + assert(ex.getMessage.toLowerCase(java.util.Locale.ROOT).contains("temporary")) + } + } + } + + test("ALTER VIEW preserves user-set TBLPROPERTIES") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_preserve " + + "TBLPROPERTIES ('mykey'='myvalue') AS " + + "SELECT x FROM spark_catalog.default.t") + sql("ALTER VIEW view_catalog.default.v_preserve AS " + + "SELECT x + 1 AS x FROM spark_catalog.default.t") + + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + val info = catalog.getStoredView(Array("default"), "v_preserve") + assert(info.properties().get("mykey") == "myvalue") + } + } + + test("CREATE VIEW stamps PROP_OWNER on the stored TableInfo") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_owner_create AS " + + "SELECT x FROM spark_catalog.default.t") + + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + val info = catalog.getStoredView(Array("default"), "v_owner_create") + // v2 CREATE VIEW stamps the current user into PROP_OWNER, matching v2 CREATE TABLE + // (via CatalogV2Util.withDefaultOwnership) and v1 CREATE VIEW (via CatalogTable.owner's + // default). Without this, the ALTER VIEW preservation test above would have nothing to + // carry forward on a v2-created view. + val owner = info.properties().get(TableCatalog.PROP_OWNER) + assert(owner != null && owner.nonEmpty, s"expected a non-empty owner, got: $owner") + } + } + + test("ALTER VIEW preserves PROP_OWNER (v1-parity)") { + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + val viewIdent = Identifier.of(Array("default"), "v_owner") + // Pre-seed a view whose stored ViewInfo carries an explicit owner. + val initialInfo = new ViewInfo.Builder() + .withSchema(new StructType().add("x", "int")) + .withQueryText("SELECT 1 AS x") + .withOwner("alice") + .withCurrentCatalog("spark_catalog") + .withCurrentNamespace(Array("default")) + .build() + catalog.createView(viewIdent, initialInfo) + try { + withTable("spark_catalog.default.t") { + Seq(2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("ALTER VIEW view_catalog.default.v_owner AS " + + "SELECT x FROM spark_catalog.default.t") + // v1 ALTER VIEW AS carries `owner` forward via `viewMeta.copy(...)`. v2 must match: + // the stored TableInfo after the ALTER should still have the original owner. + val info = catalog.getStoredView(Array("default"), "v_owner") + assert(info.properties().get(TableCatalog.PROP_OWNER) == "alice") + } + } finally { + catalog.dropTable(viewIdent) + } + } + + test("ALTER VIEW preserves SCHEMA EVOLUTION binding mode") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_evo WITH SCHEMA EVOLUTION AS " + + "SELECT x FROM spark_catalog.default.t") + sql("ALTER VIEW view_catalog.default.v_evo AS " + + "SELECT x + 1 AS x FROM spark_catalog.default.t") + + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + assert(catalog.getStoredView(Array("default"), "v_evo").schemaMode() == "EVOLUTION") + } + } + + test("ALTER VIEW re-captures the current session's SQL configs") { + withTable("spark_catalog.default.t") { + Seq("a", "b").toDF("col").write.saveAsTable("spark_catalog.default.t") + withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") { + sql("CREATE VIEW view_catalog.default.v_configs AS " + + "SELECT col FROM spark_catalog.default.t") + } + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + assert(catalog.getStoredView(Array("default"), "v_configs") + .sqlConfigs().get(SQLConf.ANSI_ENABLED.key) == "true") + + // ALTER under a different ANSI setting should replace the stored config, not merge. + withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") { + sql("ALTER VIEW view_catalog.default.v_configs AS " + + "SELECT col FROM spark_catalog.default.t WHERE col = 'b'") + } + assert(catalog.getStoredView(Array("default"), "v_configs") + .sqlConfigs().get(SQLConf.ANSI_ENABLED.key) == "false") + } + } + + test("CREATE OR REPLACE VIEW whose new body references a nonexistent table fails at " + + "analysis") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_replace_missing AS " + + "SELECT x FROM spark_catalog.default.t") + val ex = intercept[AnalysisException] { + sql("CREATE OR REPLACE VIEW view_catalog.default.v_replace_missing AS " + + "SELECT * FROM spark_catalog.default.does_not_exist") + } + assert(ex.getCondition == "TABLE_OR_VIEW_NOT_FOUND") + } + } + + test("ALTER VIEW on a catalog without ViewCatalog fails with MISSING_CATALOG_ABILITY") { + // ALTER VIEW's identifier is resolved via `UnresolvedView`, whose `viewOnly=true` path + // in `Analyzer.lookupTableOrView` rejects non-ViewCatalog catalogs up front with the + // expected error class -- before `loadTable` is even called. `TestingTableOnlyCatalog` + // happens to round-trip `default.v` as a view-typed MetadataOnlyTable, but that fixture + // is not actually consulted on this path. CREATE VIEW's capability check lives in + // `CheckViewReferences`; ALTER VIEW's lives in the analyzer gate. Both yield + // `MISSING_CATALOG_ABILITY.VIEWS`. + withSQLConf( + "spark.sql.catalog.no_view_catalog" -> classOf[TestingTableOnlyCatalog].getName) { + val ex = intercept[AnalysisException] { + sql("ALTER VIEW no_view_catalog.default.v AS SELECT 1 AS x") + } + assert(ex.getCondition == "MISSING_CATALOG_ABILITY.VIEWS") + } + } + + // --- Pure ViewCatalog (no TableCatalog mixin) --------------------------- + + test("read view from a pure ViewCatalog (no TableCatalog mixin)") { + // The analyzer's table-side lookup must skip `loadTable` entirely for catalogs that don't + // implement `TableCatalog`; otherwise `asTableCatalog` would throw + // MISSING_CATALOG_ABILITY.TABLES and the legitimate `loadView` fallback would never run. + withSQLConf( + "spark.sql.catalog.view_only" -> classOf[TestingViewOnlyCatalog].getName) { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + // The fixture stores a `pure_v` view whose body filters spark_catalog.default.t. + checkAnswer(spark.table("view_only.default.pure_v"), Seq(Row(2), Row(3))) + } + } + } + + test("ALTER VIEW on a pure ViewCatalog (no TableCatalog mixin)") { + withSQLConf( + "spark.sql.catalog.view_only" -> classOf[TestingViewOnlyCatalog].getName) { + val catalog = spark.sessionState.catalogManager.catalog("view_only") + .asInstanceOf[TestingViewOnlyCatalog] + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("ALTER VIEW view_only.default.pure_v AS " + + "SELECT x FROM spark_catalog.default.t WHERE x > 2") + assert(catalog.loadView(Identifier.of(Array("default"), "pure_v")).queryText() == + "SELECT x FROM spark_catalog.default.t WHERE x > 2") + } + } + } + + test("cyclic detection distinguishes views across multi-level namespaces") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + + // Two views whose last namespace segment collides (`inner`) but whose full multi-part + // identifiers differ. Before the `fullIdent` change both collapsed to + // `TableIdentifier(v, Some("inner"), Some("view_catalog"))` and cyclic detection would + // false-positive on a legitimate cross-namespace REPLACE. + sql("CREATE VIEW view_catalog.ns1.inner.v AS SELECT x FROM spark_catalog.default.t") + sql("CREATE VIEW view_catalog.ns2.inner.v AS " + + "SELECT x FROM view_catalog.ns1.inner.v") + // Legitimate non-cyclic REPLACE -- new body references a different view that happens to + // share the last namespace segment. Must not false-positive. + sql("CREATE OR REPLACE VIEW view_catalog.ns1.inner.v AS " + + "SELECT x FROM spark_catalog.default.t WHERE x > 1") + checkAnswer(spark.table("view_catalog.ns1.inner.v"), Seq(Row(2), Row(3))) + + // Real cycle across the two namespaces must still be caught. + val ex = intercept[AnalysisException] { + sql("CREATE OR REPLACE VIEW view_catalog.ns1.inner.v AS " + + "SELECT x FROM view_catalog.ns2.inner.v") + } + assert(ex.getCondition == "RECURSIVE_VIEW") + } + } + + test("view error messages render the full multi-level namespace") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.ns1.inner.v_err AS " + + "SELECT x FROM spark_catalog.default.t") + // Second CREATE surfaces `viewAlreadyExistsError` (via TableAlreadyExistsException from + // the catalog). Before the error signatures took `Seq[String]`, `legacyName` collapsed + // ns1.inner into just `inner` and the error said `view_catalog.inner.v_err` -- missing + // the outer `ns1` segment. + val dup = intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.ns1.inner.v_err AS " + + "SELECT x FROM spark_catalog.default.t") + } + assert(dup.getCondition == "TABLE_OR_VIEW_ALREADY_EXISTS") + assert(dup.getMessage.contains("`view_catalog`.`ns1`.`inner`.`v_err`"), + s"expected full multi-part name in error, got: ${dup.getMessage}") + + // CREATE OR REPLACE VIEW over a non-view table entry surfaces + // `unsupportedCreateOrReplaceViewOnTableError`. Pre-seed a non-view entry at a + // multi-level-namespace identifier to exercise the rendering. + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + val tblIdent = Identifier.of(Array("ns1", "inner"), "t_err") + catalog.createTable( + tblIdent, + new TableInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withTableType(TableSummary.EXTERNAL_TABLE_TYPE) + .build()) + try { + val notView = intercept[AnalysisException] { + sql("CREATE OR REPLACE VIEW view_catalog.ns1.inner.t_err AS " + + "SELECT x FROM spark_catalog.default.t") + } + assert(notView.getCondition == "EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE") + assert(notView.getMessage.contains("`view_catalog`.`ns1`.`inner`.`t_err`"), + s"expected full multi-part name in error, got: ${notView.getMessage}") + } finally { + catalog.dropTable(tblIdent) + } + + // Column-arity mismatch error. + val arity = intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.ns1.inner.v_arity (a, b) AS " + + "SELECT x FROM spark_catalog.default.t") + } + assert(arity.getMessage.contains("`view_catalog`.`ns1`.`inner`.`v_arity`"), + s"expected full multi-part name in error, got: ${arity.getMessage}") + } + } + + test("ALTER VIEW cyclic detection distinguishes views across multi-level namespaces") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + + sql("CREATE VIEW view_catalog.ns1.inner.v_alter AS " + + "SELECT x FROM spark_catalog.default.t") + sql("CREATE VIEW view_catalog.ns2.inner.v_alter AS " + + "SELECT x FROM view_catalog.ns1.inner.v_alter") + + // Legitimate non-cyclic ALTER -- new body does not reference the altered view. Before + // `fullIdent` this false-positived because the two views collapsed to the same + // TableIdentifier(v_alter, Some("inner"), Some("view_catalog")). + sql("ALTER VIEW view_catalog.ns1.inner.v_alter AS " + + "SELECT x FROM spark_catalog.default.t WHERE x > 1") + checkAnswer( + spark.table("view_catalog.ns1.inner.v_alter"), + Seq(Row(2), Row(3))) + + // Real cycle across the two namespaces must still be caught. + val ex = intercept[AnalysisException] { + sql("ALTER VIEW view_catalog.ns1.inner.v_alter AS " + + "SELECT x FROM view_catalog.ns2.inner.v_alter") + } + assert(ex.getCondition == "RECURSIVE_VIEW") + } + } + + test("temp-object reference errors render the full multi-level namespace") { + // `verifyTemporaryObjectsNotExists` / `verifyAutoGeneratedAliasesNotExists` used to take a + // `TableIdentifier` built via `asLegacyTableIdentifier`, which collapses multi-level + // namespaces to the last segment -- so a temp-function reference on + // `view_catalog.ns1.inner.v_tempfn` produced an error naming + // `view_catalog.inner.v_tempfn` and dropped the `ns1` middle segment. Post-migration the + // errors render the full multi-part name. + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + spark.udf.register("temp_udf_multi", (i: Int) => i + 1) + val ex = intercept[AnalysisException] { + sql("CREATE VIEW view_catalog.ns1.inner.v_tempfn AS " + + "SELECT temp_udf_multi(x) FROM spark_catalog.default.t") + } + assert(ex.getCondition == "INVALID_TEMP_OBJ_REFERENCE") + assert(ex.getMessage.contains("`view_catalog`.`ns1`.`inner`.`v_tempfn`"), + s"expected full multi-part name, got: ${ex.getMessage}") + } + } + + // --- Follow-up-blocked view DDL / inspection on a non-session v2 catalog ------------ + // These plans don't have a dedicated v2 strategy yet (tracked for a follow-up PR). We pin + // the current failure mode -- UNSUPPORTED_FEATURE.TABLE_OPERATION with a statement-specific + // operation string -- so a future generic "no plan found" regression would surface here + // rather than silently degrading the UX. + + private def seedV2View(name: String): Unit = { + sql(s"CREATE VIEW view_catalog.default.$name AS SELECT 1 AS x") + } + + private def assertUnsupportedViewOp(statement: String): Unit = { + val ex = intercept[AnalysisException](sql(statement)) + assert(ex.getCondition == "UNSUPPORTED_FEATURE.TABLE_OPERATION", s"got ${ex.getCondition}") + } + + test("ALTER VIEW ... SET TBLPROPERTIES on a v2 view is rejected") { + seedV2View("v_set_props") + assertUnsupportedViewOp( + "ALTER VIEW view_catalog.default.v_set_props SET TBLPROPERTIES ('k' = 'v')") + } + + test("ALTER VIEW ... UNSET TBLPROPERTIES on a v2 view is rejected") { + seedV2View("v_unset_props") + assertUnsupportedViewOp( + "ALTER VIEW view_catalog.default.v_unset_props UNSET TBLPROPERTIES ('k')") + } + + test("ALTER VIEW ... WITH SCHEMA on a v2 view is rejected") { + seedV2View("v_schema_binding") + assertUnsupportedViewOp( + "ALTER VIEW view_catalog.default.v_schema_binding WITH SCHEMA EVOLUTION") + } + + test("ALTER VIEW ... RENAME TO on a v2 view is rejected") { + seedV2View("v_rename") + assertUnsupportedViewOp( + "ALTER VIEW view_catalog.default.v_rename RENAME TO view_catalog.default.v_renamed") + } + + test("SHOW CREATE TABLE on a v2 view is rejected") { + seedV2View("v_show_create") + assertUnsupportedViewOp("SHOW CREATE TABLE view_catalog.default.v_show_create") + } + + test("SHOW TBLPROPERTIES on a v2 view is rejected") { + seedV2View("v_show_props") + assertUnsupportedViewOp("SHOW TBLPROPERTIES view_catalog.default.v_show_props") + } + + test("SHOW COLUMNS on a v2 view is rejected") { + seedV2View("v_show_cols") + assertUnsupportedViewOp("SHOW COLUMNS IN view_catalog.default.v_show_cols") + } + + test("DESCRIBE TABLE on a v2 view is rejected") { + seedV2View("v_describe") + assertUnsupportedViewOp("DESCRIBE TABLE view_catalog.default.v_describe") + } + + test("DESCRIBE TABLE ... COLUMN on a v2 view is rejected") { + seedV2View("v_describe_col") + // Column resolution against a v2 view's output isn't wired up yet, so the analyzer fails + // with UNRESOLVED_COLUMN before reaching the planner. That's still a clean + // AnalysisException (not a generic "no plan found"), which is the pin we care about. + intercept[AnalysisException]( + sql("DESCRIBE TABLE view_catalog.default.v_describe_col x")) + } + + // These plans reach `DataSourceV2Strategy` with a `ResolvedPersistentView` child on a + // non-session v2 view (because `ResolvedV1TableOrViewIdentifier` now skips non-session views). + // Without explicit pins they would hit `QueryPlanner`'s `assert(pruned.hasNext, "No plan for + // ...")` and surface a raw AssertionError. Pin each to UNSUPPORTED_FEATURE.TABLE_OPERATION. + + test("REFRESH TABLE on a v2 view is rejected") { + seedV2View("v_refresh") + assertUnsupportedViewOp("REFRESH TABLE view_catalog.default.v_refresh") + } + + test("ANALYZE TABLE on a v2 view is rejected") { + seedV2View("v_analyze") + assertUnsupportedViewOp( + "ANALYZE TABLE view_catalog.default.v_analyze COMPUTE STATISTICS") + } + + test("ANALYZE TABLE ... FOR COLUMNS on a v2 view is rejected") { + seedV2View("v_analyze_cols") + assertUnsupportedViewOp( + "ANALYZE TABLE view_catalog.default.v_analyze_cols COMPUTE STATISTICS FOR COLUMNS x") + } + + // --- DROP VIEW on a v2 catalog -------------------------------- + + test("DROP VIEW on a ViewCatalog drops the view") { + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_drop AS " + + "SELECT x FROM spark_catalog.default.t") + assert(catalog.viewExists(Identifier.of(Array("default"), "v_drop"))) + sql("DROP VIEW view_catalog.default.v_drop") + assert(!catalog.viewExists(Identifier.of(Array("default"), "v_drop"))) + } + } + + test("DROP VIEW IF EXISTS on a v2 catalog is a no-op when the view is missing") { + // Exercises the `ifExists=true` path -- DropViewExec should not throw when the view + // doesn't exist on a ViewCatalog. + sql("DROP VIEW IF EXISTS view_catalog.default.v_never_existed") + } + + test("DROP VIEW on a non-view table entry is rejected (v1-parity)") { + // v1 `DropTableCommand(isView = true)` rejects a non-view target via + // `wrongCommandForObjectTypeError`. The v2 path must also refuse -- otherwise + // `DROP VIEW view_catalog.default.` would silently destroy the table's entry. + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + val tableIdent = Identifier.of(Array("default"), "t_not_a_view") + catalog.createTable( + tableIdent, + new TableInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withTableType(TableSummary.EXTERNAL_TABLE_TYPE) + .build()) + try { + val ex = intercept[AnalysisException] { + sql("DROP VIEW view_catalog.default.t_not_a_view") + } + assert(ex.getCondition == "EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE") + // The table entry must still be there -- DROP VIEW did not destroy it. + assert(catalog.tableExists(tableIdent)) + } finally { + catalog.dropTable(tableIdent) + } + } + + test("DROP VIEW on a catalog without ViewCatalog is rejected") { + withSQLConf( + "spark.sql.catalog.no_view_catalog" -> classOf[TestingTableOnlyCatalog].getName) { + val ex = intercept[AnalysisException] { + sql("DROP VIEW no_view_catalog.default.v") + } + // Preserves the pre-PR error surface for non-ViewCatalog catalogs. + assert(ex.getMessage.toLowerCase(java.util.Locale.ROOT).contains("views")) + } + } + + // --- SHOW TABLES / SHOW VIEWS on a v2 catalog -------------------------------- + + private def seedV2Table(name: String): Unit = { + val catalog = spark.sessionState.catalogManager.catalog("view_catalog") + .asInstanceOf[TestingRelationCatalog] + catalog.createTable( + Identifier.of(Array("default"), name), + new TableInfo.Builder() + .withSchema(new StructType().add("x", "int")) + .withTableType(TableSummary.EXTERNAL_TABLE_TYPE) + .build()) + } + + test("SHOW TABLES on a v2 catalog returns only tables") { + // Per the new `TableCatalog.listTables` contract, SHOW TABLES returns table identifiers + // only -- views (in mixed catalogs) are listed via SHOW VIEWS / `ViewCatalog.listViews`. + // This is an intentional divergence from v1 SHOW TABLES (which includes both tables and + // views in a single listing); v2 catalogs separate the two so callers can target either + // kind without filtering. + seedV2View("v_in_show_tables") + seedV2Table("t_in_show_tables") + val rows = sql("SHOW TABLES IN view_catalog.default").collect() + val names = rows.map(_.getString(1)).toSet + assert(names.contains("t_in_show_tables"), s"table missing from SHOW TABLES: $names") + assert(!names.contains("v_in_show_tables"), s"view leaked into SHOW TABLES: $names") + rows.foreach(r => assert(!r.getBoolean(2), s"isTemporary must be false: $r")) + } + + test("SHOW VIEWS on a v2 catalog returns only views") { + seedV2View("v_in_show_views") + seedV2Table("t_not_in_show_views") + val rows = sql("SHOW VIEWS IN view_catalog.default").collect() + val names = rows.map(_.getString(1)).toSet + assert(names.contains("v_in_show_views"), s"view missing: $names") + assert(!names.contains("t_not_in_show_views"), + s"non-view leaked into SHOW VIEWS: $names") + rows.foreach(r => assert(!r.getBoolean(2), s"isTemporary must be false for v2: $r")) + } + + test("SHOW VIEWS with LIKE pattern filters on the view name") { + seedV2View("v_foo") + seedV2View("v_bar") + val rows = sql("SHOW VIEWS IN view_catalog.default LIKE 'v_foo'").collect() + val names = rows.map(_.getString(1)).toSet + assert(names == Set("v_foo"), s"expected only v_foo, got $names") + } + + test("SHOW VIEWS on a catalog without ViewCatalog is rejected") { + withSQLConf( + "spark.sql.catalog.no_view_catalog" -> classOf[TestingTableOnlyCatalog].getName) { + val ex = intercept[AnalysisException] { + sql("SHOW VIEWS IN no_view_catalog.default") + } + assert(ex.getCondition == "MISSING_CATALOG_ABILITY.VIEWS") + } + } + + test("ALTER VIEW detects cyclic view references") { + withTable("spark_catalog.default.t") { + Seq(1, 2, 3).toDF("x").write.saveAsTable("spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_alter_cycle_a AS " + + "SELECT x FROM spark_catalog.default.t") + sql("CREATE VIEW view_catalog.default.v_alter_cycle_b AS " + + "SELECT x FROM view_catalog.default.v_alter_cycle_a") + val ex = intercept[AnalysisException] { + sql("ALTER VIEW view_catalog.default.v_alter_cycle_a AS " + + "SELECT x FROM view_catalog.default.v_alter_cycle_b") + } + assert(ex.getCondition == "RECURSIVE_VIEW") + } + } +} + +/** + * A [[RelationCatalog]]: round-trips [[MetadataOnlyTable]] for created views and tables and + * exposes a few canned read-only view fixtures (`test_view`, `test_unqualified_view`, + * `test_unqualified_multi`, plus an unqualified-target view at `ns1.ns2.t`) used by the + * view-read tests. Entries created via `createTable` / `createView` are distinguished by the + * stored value's runtime type (ViewInfo vs TableInfo). The single-RPC perf entry point + * [[loadRelation]] returns either kind; [[loadTable]] is tables-only per the + * [[TableCatalog#loadTable]] contract. + */ +class TestingRelationCatalog extends RelationCatalog { + + // Holds entries (views and tables) created via createTable / createView within the session. + // Keyed by (namespace, name); the stored value's runtime type (ViewInfo vs TableInfo) + // distinguishes views from tables. Mixed-catalog: shared identifier namespace per the + // RelationCatalog contract. + private val createdViews = + new java.util.concurrent.ConcurrentHashMap[(Seq[String], String), TableInfo]() + + // Canned read-only view fixtures, exposed only via the perf path (loadRelation). loadView + // does not need to expose them because the resolver routes RelationCatalog reads through + // loadRelation. + private def fixtureView(ident: Identifier): Option[ViewInfo] = ident.name() match { + case "test_view" => + Some(new ViewInfo.Builder() + .withSchema(new StructType().add("col", "string").add("i", "int")) + .withQueryText( + "SELECT col, col::int AS i FROM spark_catalog.default.t WHERE col = 'b'") + .withSqlConfigs(java.util.Collections.singletonMap( + SQLConf.ANSI_ENABLED.key, (ident.namespace().head == "ansi").toString)) + .build()) + case "test_unqualified_view" => + Some(new ViewInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withQueryText("SELECT col FROM t WHERE col = 'b'") + .withCurrentCatalog("spark_catalog") + .withCurrentNamespace(Array("default")) + .build()) + case "test_unqualified_multi" => + // View whose captured catalog+namespace is view_catalog.ns1.ns2 (two-part). The + // unqualified `t` in the body must resolve via that captured context to + // view_catalog.ns1.ns2.t, which this catalog also serves (see `t` case below). + Some(new ViewInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withQueryText("SELECT col FROM t") + .withCurrentCatalog("view_catalog") + .withCurrentNamespace(Array("ns1", "ns2")) + .build()) + case "t" if ident.namespace().toSeq == Seq("ns1", "ns2") => + // Target of test_unqualified_multi's unqualified reference. Self-contained view so + // the test doesn't need external data. + Some(new ViewInfo.Builder() + .withSchema(new StructType().add("col", "string")) + .withQueryText("SELECT 'multi' AS col") + .build()) + case _ => None + } + + override def loadRelation(ident: Identifier): Table = { + // Single-RPC perf path: returns tables AND views (as MetadataOnlyTable). Stored entries + // win over fixture views (the fixture namespace is read-only and disjoint from + // createdViews in practice). loadTable, loadView, tableExists, viewExists all derive + // from this via the RelationCatalog default impls. + val key = (ident.namespace().toSeq, ident.name()) + Option(createdViews.get(key)) + .orElse(fixtureView(ident)) + .map(new MetadataOnlyTable(_, ident.toString)) + .getOrElse(throw new NoSuchTableException(ident)) + } + + override def createTable(ident: Identifier, info: TableInfo): Table = { + // Mixed-catalog contract: createTable rejects when a view sits at ident with + // TableAlreadyExistsException. The shared `createdViews` keyspace makes `putIfAbsent` + // throw uniformly for both table-at-ident and view-at-ident collisions. + val key = (ident.namespace().toSeq, ident.name()) + if (createdViews.putIfAbsent(key, info) != null) { + throw new TableAlreadyExistsException(ident) + } + new MetadataOnlyTable(info, ident.toString) + } + + /** Test-only accessor: returns the stored TableInfo (table or view) for the identifier. */ + def getStoredInfo(namespace: Array[String], name: String): TableInfo = { + Option(createdViews.get((namespace.toSeq, name))).getOrElse { + throw new NoSuchTableException(Identifier.of(namespace, name)) + } + } + + /** Test-only accessor: returns the stored ViewInfo; fails if the entry is not a view. */ + def getStoredView(namespace: Array[String], name: String): ViewInfo = getStoredInfo( + namespace, name) match { + case v: ViewInfo => v + case _ => throw new IllegalStateException( + s"stored entry at ${namespace.mkString(".")}.$name is not a view") + } + + override def alterTable(ident: Identifier, changes: TableChange*): Table = { + throw new RuntimeException("shouldn't be called") + } + override def dropTable(ident: Identifier): Boolean = { + val key = (ident.namespace().toSeq, ident.name()) + val existing = createdViews.get(key) + if (existing == null || existing.isInstanceOf[ViewInfo]) return false + createdViews.remove(key) != null + } + override def renameTable(oldIdent: Identifier, newIdent: Identifier): Unit = { + throw new RuntimeException("shouldn't be called") + } + override def listTables(namespace: Array[String]): Array[Identifier] = { + // Tables only -- views are listed via ViewCatalog.listViews per the new contract. + val targetNs = namespace.toSeq + val ids = new java.util.ArrayList[Identifier]() + createdViews.forEach { (key, info) => + if (key._1 == targetNs && !info.isInstanceOf[ViewInfo]) { + ids.add(Identifier.of(key._1.toArray, key._2)) + } + } + ids.toArray(new Array[Identifier](0)) + } + + // ViewCatalog methods. Storage is shared with TableCatalog (mixed-catalog pattern). + + override def listViews(namespace: Array[String]): Array[Identifier] = { + val targetNs = namespace.toSeq + val ids = new java.util.ArrayList[Identifier]() + createdViews.forEach { (key, info) => + if (key._1 == targetNs && info.isInstanceOf[ViewInfo]) { + ids.add(Identifier.of(key._1.toArray, key._2)) + } + } + ids.toArray(new Array[Identifier](0)) + } + + override def createView(ident: Identifier, info: ViewInfo): ViewInfo = { + val key = (ident.namespace().toSeq, ident.name()) + if (createdViews.putIfAbsent(key, info) != null) { + throw new ViewAlreadyExistsException(ident) + } + info + } + + override def replaceView(ident: Identifier, info: ViewInfo): ViewInfo = { + val key = (ident.namespace().toSeq, ident.name()) + val existing = createdViews.get(key) + if (existing == null || !existing.isInstanceOf[ViewInfo]) { + throw new NoSuchViewException(ident) + } + createdViews.put(key, info) + info + } + + override def dropView(ident: Identifier): Boolean = { + val key = (ident.namespace().toSeq, ident.name()) + val existing = createdViews.get(key) + if (existing == null || !existing.isInstanceOf[ViewInfo]) return false + createdViews.remove(key) != null + } + + private var catalogName = "" + override def initialize(name: String, options: CaseInsensitiveStringMap): Unit = { + catalogName = name + } + override def name(): String = catalogName +} + +/** + * A v2 catalog that does not implement ViewCatalog. Used by capability-gate tests: the gate + * fires in `Analyzer.lookupTableOrView(viewOnly=true)` for ALTER VIEW and in + * [[CheckViewReferences]] for CREATE VIEW -- in both cases before `loadTable` is called -- + * so this catalog's content is intentionally empty. + */ +class TestingTableOnlyCatalog extends TableCatalog { + override def loadTable(ident: Identifier): Table = throw new NoSuchTableException(ident) + + override def alterTable(ident: Identifier, changes: TableChange*): Table = + throw new RuntimeException("shouldn't be called") + override def dropTable(ident: Identifier): Boolean = false + override def renameTable(oldIdent: Identifier, newIdent: Identifier): Unit = + throw new RuntimeException("shouldn't be called") + override def listTables(namespace: Array[String]): Array[Identifier] = Array.empty + private var catalogName = "" + override def initialize(name: String, options: CaseInsensitiveStringMap): Unit = { + catalogName = name + } + override def name(): String = catalogName +} + +/** + * A pure [[ViewCatalog]] (no [[TableCatalog]] mixin). Used to exercise that the analyzer's + * resolution paths skip the `loadTable` step and fall through to `loadView` for catalogs that + * cannot host tables. Pre-seeds a single mutable view at `default.pure_v` so the read and + * ALTER VIEW tests can both reach it. + */ +class TestingViewOnlyCatalog extends ViewCatalog { + private val store = + new java.util.concurrent.ConcurrentHashMap[(Seq[String], String), ViewInfo]() + + // Seeded on first `initialize`. Filters `spark_catalog.default.t` so the read test can + // assert deterministic output. ALTER VIEW tests overwrite it via `replaceView`. + private def seedDefault(): Unit = { + val key = (Seq("default"), "pure_v") + if (!store.containsKey(key)) { + val info = new ViewInfo.Builder() + .withSchema(new StructType().add("x", "int")) + .withQueryText("SELECT x FROM spark_catalog.default.t WHERE x > 1") + .build() + store.put(key, info) + } + } + + override def listViews(namespace: Array[String]): Array[Identifier] = { + val target = namespace.toSeq + val ids = new java.util.ArrayList[Identifier]() + store.forEach { (key, _) => + if (key._1 == target) ids.add(Identifier.of(key._1.toArray, key._2)) + } + ids.toArray(new Array[Identifier](0)) + } + + override def loadView(ident: Identifier): ViewInfo = { + val key = (ident.namespace().toSeq, ident.name()) + Option(store.get(key)).getOrElse(throw new NoSuchViewException(ident)) + } + + override def createView(ident: Identifier, info: ViewInfo): ViewInfo = { + val key = (ident.namespace().toSeq, ident.name()) + if (store.putIfAbsent(key, info) != null) { + throw new ViewAlreadyExistsException(ident) + } + info + } + + override def replaceView(ident: Identifier, info: ViewInfo): ViewInfo = { + val key = (ident.namespace().toSeq, ident.name()) + if (!store.containsKey(key)) throw new NoSuchViewException(ident) + store.put(key, info) + info + } + + override def dropView(ident: Identifier): Boolean = { + val key = (ident.namespace().toSeq, ident.name()) + store.remove(key) != null + } + + private var catalogName = "" + override def initialize(name: String, options: CaseInsensitiveStringMap): Unit = { + catalogName = name + seedDefault() + } + override def name(): String = catalogName +} diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala index d2cc342f48112..d1dc9c282829f 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala @@ -2966,13 +2966,13 @@ class DataSourceV2SQLSuiteV1Filter } } - test("View commands are not supported in v2 catalogs") { + test("View commands are not supported in v2 catalogs that don't implement ViewCatalog") { def validateViewCommand(sqlStatement: String): Unit = { val e = analysisException(sqlStatement) checkError( e, - condition = "UNSUPPORTED_FEATURE.CATALOG_OPERATION", - parameters = Map("catalogName" -> "`testcat`", "operation" -> "views")) + condition = "MISSING_CATALOG_ABILITY.VIEWS", + parameters = Map("plugin" -> "testcat")) } validateViewCommand("DROP VIEW testcat.v") diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala index 89fb6eca223ee..b564cad0fe9c8 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala @@ -778,8 +778,8 @@ class PlanResolutionSuite extends SharedSparkSession with AnalysisTest { } checkError( e, - condition = "UNSUPPORTED_FEATURE.CATALOG_OPERATION", - parameters = Map("catalogName" -> "`testcat`", "operation" -> "views")) + condition = "MISSING_CATALOG_ABILITY.VIEWS", + parameters = Map("plugin" -> "testcat")) } // ALTER VIEW view_name SET TBLPROPERTIES ('comment' = new_comment); diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DropTableSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DropTableSuite.scala index ffc2c6c679a8b..0e5cbb861d05d 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DropTableSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DropTableSuite.scala @@ -41,6 +41,13 @@ class DropTableSuite extends command.DropTableSuiteBase with CommandSuiteBase { } } + test("DROP TABLE IF EXISTS ... PURGE on a missing table is a no-op") { + // The default TableCatalog.purgeTable throws unconditionally, so without an upfront + // existence guard `IF EXISTS` would surface UNSUPPORTED_FEATURE.PURGE_TABLE for missing + // tables -- defeating the IF EXISTS contract on catalogs that do not support purge. + sql(s"DROP TABLE IF EXISTS $catalog.ns.never_existed PURGE") + } + test("table qualified with the session catalog name") { withSQLConf( V2_SESSION_CATALOG_IMPLEMENTATION.key -> classOf[InMemoryTableSessionCatalog].getName) { From 34159e5a7bfd707919237a8e29e958cce5d140d7 Mon Sep 17 00:00:00 2001 From: Juliusz Sompolski <25019163+juliuszsompolski@users.noreply.github.com> Date: Tue, 28 Apr 2026 21:32:18 +0800 Subject: [PATCH 003/286] [SPARK-56509][SQL] SparkSQL Last Attempt Metrics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? This PR introduces Last Attempt Accumulators — accumulators that track metric values aggregated across the "last execution" that produced the values, discarding values from earlier attempts that have been recomputed due to stage retries or AQE replanning. #### The problem Regular Spark accumulators sum up values from all task attempts, including retried ones. When a stage retry occurs — due to executor loss, fetch failures, or AQE replanning — the accumulator value includes contributions from both the original execution and the retry. This makes it impossible to determine the "true" metric value of the final successful execution. For example, if a stage processes 1000 rows and is retried once, `numOutputRows` reports 2000 instead of 1000. This is a fundamental limitation when accumulators are used for post-execution correctness checking (e.g. verifying that the number of rows written matches expectations), or when accurate execution statistics need to be reported. #### The solution `LastAttemptAccumulator` is a trait that can be mixed into any `AccumulatorV2` subclass. It hooks into `DAGScheduler.updateAccumulators` to receive per-task stage and attempt metadata alongside the regular accumulator merge. It tracks per-RDD, per-partition partial values tagged with `(stageId, stageAttemptId, taskAttemptNumber)`, and when queried, aggregates only the values from the latest attempt of each partition. #### API The core trait `LastAttemptAccumulator[IN, OUT, PARTIAL]` provides several query methods to retrieve the last attempt value, depending on what scope is of interest: - **`lastAttemptValueForRDDId(rddId)`** / **`lastAttemptValueForRDDIds(rddIds)`** — Value from specific RDD(s). Useful when the caller knows which RDD(s) the metric was used in. - **`lastAttemptValueForAllRDDs()`** — Value aggregated from all RDDs that contributed to this accumulator. Useful when only one execution used the accumulator. - **`lastAttemptValueForHighestRDDId()`** — Value from the RDD with the highest ID, which corresponds to the most recent execution. A simple heuristic that works well for single-use metrics in SQL plans. - **`lastAttemptValueForRDDScopes(rddScopeIds)`** — Value from RDDs matching specific `RDDOperationScope` IDs, enabling tracking by SparkPlan node. All query methods return `Option[OUT]` — `None` means the last attempt value cannot be determined (e.g. the accumulator was updated both from tasks and directly on the driver, or an internal consistency check failed). The SQL extension `SQLLastAttemptAccumulator` adds Dataset-aware tracking: - **`lastAttemptValueForDataset(ds)`** / **`lastAttemptValueForQueryExecution(qe)`** — Value from the execution of a specific Dataset or QueryExecution. This works by extracting `RDDOperationScope` IDs from the physical plan to identify which RDDs belong to which SparkPlan nodes, and is aware of exchanges, subqueries, broadcast joins, and WholeStageCodegen fallbacks. A ready-to-use concrete implementation `SQLLastAttemptMetric` (a `SQLMetric` with `SQLLastAttemptAccumulator` mixed in) can be created via `SQLLastAttemptMetrics.createMetric(sparkContext, name)`. #### Scoping and edge cases handled - **Stage retries**: Tracks `(stageId, stageAttemptId, taskAttemptNumber)` tuples, keeps only the latest. - **AQE replanning**: New plans create new RDDs with new scopes; old cancelled plan values are naturally excluded when querying by scope. - **Repeated Dataset execution**: Same `executedPlan` and RDDs are reused; the accumulator detects this and doesn't double-count. - **Partial execution** (`take`/`limit`): Only computed partitions are aggregated. - **Driver-side updates** (e.g. `ConvertToLocalRelation` optimizer folding): Tracked separately, scoped to `QueryExecution.id`. Bails out if mixed with task-side updates. - **WholeStageCodegen fallback**: Accounts for both the codegen wrapper's and the child's RDD scope, since fallback changes which scope the execution runs in. #### Testing infrastructure This PR also adds two testing-only mechanisms: - `INJECT_SHUFFLE_FETCH_FAILURES`: a config that injects invalid BlockManager locations for the first stage attempt of shuffle map tasks, forcing stage retries. - `AQETestHelper.withForcedCancellation`: triggers forced AQE replanning after the first stage materializes with non-zero metric values, causing the plan to be discarded and re-run. Together these enable testing Last Attempt Accumulators under both stage retry and AQE replanning conditions. ### Why are the changes needed? Applications and future work: - **DML statistics**: Reporting correct rows written/updated/deleted after `INSERT`, `UPDATE`, `DELETE`, `MERGE` operations, even when stage retries occurred. - **Observability**: Any metric that should reflect the actual work done by the final successful execution, not cumulative work across all attempts. We could consider reporting metrics like "num output rows" of various operators in Spark UI using SLAM. ### Does this PR introduce _any_ user-facing change? This adds internal infrastructure. `LastAttemptAccumulator` and `SQLLastAttemptMetric` are internal APIs. No existing accumulator behavior is changed. There is a slight change in RDDOperationScope names. SparkPlan used to create scopes with an autogenerated name (`RDDOperationScope.nextScopeId`. Now they get a scope name assigned as `spark_plan_{id}` where `id` is the SparkPlan.id, which is also a globally unique, monotonically incrementing id. This is visible in the SparkUI, and makes it more readable to link the RDD operation with the SparkPlan that triggered it. ### How was this patch tested? - `SQLLastAttemptMetricUnitSuite` — Unit tests for serialization, copy, and `mergeLastAttempt` with various attempt orderings. - `SQLLastAttemptMetricIntegrationSuite` — Integration tests with RDDs and Datasets covering: single/multi stage execution, `take`/`coalesce`, driver-side updates, `ConvertToLocalRelation` optimizer folding, `BroadcastNestedLoopJoin` double execution, AQE coalesced shuffle partitions, and `WholeStageCodegenExec` fallback. - `SQLLastAttemptMetricIntegrationSuiteWithStageRetries` — Reruns all integration tests with `INJECT_SHUFFLE_FETCH_FAILURES` enabled to verify correctness under stage retries. - `SQLLastAttemptMetricPlanShapesSuite` — Tests various physical plan shapes (simple plans, subqueries, shuffle/broadcast/reused exchanges, cached plans, checkpointed plans) across an AQE on/off, stage retry on/off, and AQE replanning on/off matrix. - `MetricsFailureInjectionSuite` — Tests the AQE forced cancellation and shuffle fetch failure injection mechanisms with multi-stage queries, verifying that both regular metrics and SLAM metrics behave correctly under retries and replanning. ### Was this patch authored or co-authored using generative AI tooling? Code artisinally crafted by a human, some refactoring and applying review comments by Claude. Generated-by: Claude Code (claude-opus-4-6) Closes #55371 from juliuszsompolski/spark-last-attempt-metrics. Lead-authored-by: Juliusz Sompolski <25019163+juliuszsompolski@users.noreply.github.com> Co-authored-by: Juliusz Sompolski Signed-off-by: Wenchen Fan --- .../org/apache/spark/internal/LogKeys.java | 4 + .../org/apache/spark/internal/Logging.scala | 5 + .../scala/org/apache/spark/SparkContext.scala | 2 + .../apache/spark/internal/config/Tests.scala | 7 + .../apache/spark/rdd/RDDOperationScope.scala | 22 +- .../apache/spark/scheduler/DAGScheduler.scala | 18 + .../spark/util/LastAttemptAccumulator.scala | 867 ++++++++++++++++++ .../apache/spark/sql/classic/Dataset.scala | 8 +- .../spark/sql/execution/QueryExecution.scala | 27 +- .../spark/sql/execution/SparkPlan.scala | 20 +- .../execution/adaptive/AQETestHelper.scala | 79 ++ .../adaptive/AdaptiveSparkPlanExec.scala | 16 +- .../adaptive/AdaptiveSparkPlanHelper.scala | 17 + .../exchange/ShuffleExchangeExec.scala | 27 +- .../metric/SQLLastAttemptAccumulator.scala | 435 +++++++++ .../metric/SQLLastAttemptMetric.scala | 88 ++ .../sql/execution/metric/SQLMetrics.scala | 6 +- .../org/apache/spark/sql/QueryTest.scala | 22 + .../metric/MetricsFailureInjectionSuite.scala | 364 ++++++++ ...SQLLastAttemptMetricIntegrationSuite.scala | 705 ++++++++++++++ .../SQLLastAttemptMetricPlanShapesSuite.scala | 490 ++++++++++ .../SQLLastAttemptMetricUnitSuite.scala | 188 ++++ .../metric/SQLMetricsTestUtils.scala | 24 + .../configs-without-binding-policy-exceptions | 1 + 24 files changed, 3417 insertions(+), 25 deletions(-) create mode 100644 core/src/main/scala/org/apache/spark/util/LastAttemptAccumulator.scala create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQETestHelper.scala create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetric.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/metric/MetricsFailureInjectionSuite.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricIntegrationSuite.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricPlanShapesSuite.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricUnitSuite.scala diff --git a/common/utils-java/src/main/java/org/apache/spark/internal/LogKeys.java b/common/utils-java/src/main/java/org/apache/spark/internal/LogKeys.java index d8238912aec63..8b2cefc1fe5ac 100644 --- a/common/utils-java/src/main/java/org/apache/spark/internal/LogKeys.java +++ b/common/utils-java/src/main/java/org/apache/spark/internal/LogKeys.java @@ -331,6 +331,10 @@ public enum LogKeys implements LogKey { LABEL_COLUMN, LARGEST_CLUSTER_INDEX, LAST_ACCESS_TIME, + LAST_ATTEMPT_ACC_INVALIDATE, + LAST_ATTEMPT_ACC_SYSTEM_METRIC, + LAST_ATTEMPT_ACC_UNEXPECTED_REASON, + LAST_ATTEMPT_ACC_USER_METRIC, LAST_COMMITTED_CHECKPOINT_ID, LAST_COMMIT_BASED_CHECKPOINT_ID, LAST_SCAN_TIME, diff --git a/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala b/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala index 810bdabebb38a..3fac57dbe5dda 100644 --- a/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala +++ b/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala @@ -95,6 +95,11 @@ class LogEntry(messageWithContext: => MessageWithContext) { def message: String = cachedMessageWithContext.message def context: java.util.Map[String, String] = cachedMessageWithContext.context + + def +(other: LogEntry): LogEntry = { + val combined = cachedMessageWithContext + other.cachedMessageWithContext + new LogEntry(combined) + } } /** diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala index fad9bb522ad92..0262144490ce8 100644 --- a/core/src/main/scala/org/apache/spark/SparkContext.scala +++ b/core/src/main/scala/org/apache/spark/SparkContext.scala @@ -3152,6 +3152,8 @@ object SparkContext extends Logging { private[spark] val RDD_SCOPE_KEY = "spark.rdd.scope" private[spark] val RDD_SCOPE_NO_OVERRIDE_KEY = "spark.rdd.scope.noOverride" private[spark] val SQL_EXECUTION_ID_KEY = "spark.sql.execution.id" + private[spark] val DATASET_QUERY_EXECUTION_ID_KEY = + "spark.sql.dataset.queryExecution.id" /** * Executor id for the driver. In earlier versions of Spark, this was ``, but this was diff --git a/core/src/main/scala/org/apache/spark/internal/config/Tests.scala b/core/src/main/scala/org/apache/spark/internal/config/Tests.scala index 98b80317db982..8ecb14be1dfb8 100644 --- a/core/src/main/scala/org/apache/spark/internal/config/Tests.scala +++ b/core/src/main/scala/org/apache/spark/internal/config/Tests.scala @@ -39,6 +39,13 @@ private[spark] object Tests { .booleanConf .createOptional + val INJECT_SHUFFLE_FETCH_FAILURES = + ConfigBuilder("spark.testing.injectShuffleFetchFailures") + .doc("Injecting fetch failures for shuffle stages by providing an invalid BlockManager " + + "location for the first stage attempt. Testing only flag!") + .booleanConf + .createWithDefault(false) + val TEST_NO_STAGE_RETRY = ConfigBuilder("spark.test.noStageRetry") .version("1.2.0") .booleanConf diff --git a/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala b/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala index 49c259999a471..675c44153cd4d 100644 --- a/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala +++ b/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala @@ -130,6 +130,22 @@ private[spark] object RDDOperationScope extends Logging { name: String, allowNesting: Boolean, ignoreParent: Boolean)(body: => T): T = { + withScope(sc, name, allowNesting, ignoreParent, + nextScopeId().toString)(body) + } + + /** + * Execute the given body such that all RDDs created in this body + * will have the same scope, with an explicit scope ID. + * + * Note: Return statements are NOT allowed in body. + */ + private[spark] def withScope[T]( + sc: SparkContext, + name: String, + allowNesting: Boolean, + ignoreParent: Boolean, + rddScopeId: String)(body: => T): T = { // Save the old scope to restore it later val scopeKey = SparkContext.RDD_SCOPE_KEY val noOverrideKey = SparkContext.RDD_SCOPE_NO_OVERRIDE_KEY @@ -139,10 +155,12 @@ private[spark] object RDDOperationScope extends Logging { try { if (ignoreParent) { // Ignore all parent settings and scopes and start afresh with our own root scope - sc.setLocalProperty(scopeKey, new RDDOperationScope(name).toJson) + sc.setLocalProperty(scopeKey, + new RDDOperationScope(name, None, rddScopeId).toJson) } else if (sc.getLocalProperty(noOverrideKey) == null) { // Otherwise, set the scope only if the higher level caller allows us to do so - sc.setLocalProperty(scopeKey, new RDDOperationScope(name, oldScope).toJson) + sc.setLocalProperty(scopeKey, + new RDDOperationScope(name, oldScope, rddScopeId).toJson) } // Optionally disallow the child body to override our scope if (!allowNesting) { diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala index 5fbd160bc683b..f3958bfddec95 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala @@ -1858,6 +1858,11 @@ private[spark] class DAGScheduler( throw SparkCoreErrors.accessNonExistentAccumulatorError(id) } acc.merge(updates.asInstanceOf[AccumulatorV2[Any, Any]]) + if (acc.isInstanceOf[LastAttemptAccumulator[_, _, _]]) { + acc.asInstanceOf[LastAttemptAccumulator[_, _, _]].mergeLastAttempt( + updates, stage.rdd, event.taskInfo, + task.stageId, task.stageAttemptId, task.localProperties) + } // To avoid UI cruft, ignore cases where value wasn't updated if (acc.name.isDefined && !updates.isZero) { stage.latestInfo.accumulables(id) = acc.toInfo(None, Some(acc.value)) @@ -2333,6 +2338,19 @@ private[spark] class DAGScheduler( // The epoch of the task is acceptable (i.e., the task was launched after the most // recent failure we're aware of for the executor), so mark the task's output as // available. + // For testing purposes, inject fetch failures controlled from the driver-side by + // supplying an invalid location. + if (Utils.isTesting && + sc.conf.get(config.Tests.INJECT_SHUFFLE_FETCH_FAILURES) && + task.stageAttemptId == 0) { + val currentLocation = status.location + val invalidLocation = BlockManagerId( + execId = BlockManagerId.INVALID_EXECUTOR_ID, + host = currentLocation.host, + port = currentLocation.port, + topologyInfo = currentLocation.topologyInfo) + status.updateLocation(invalidLocation) + } val isChecksumMismatched = mapOutputTracker.registerMapOutput( shuffleStage.shuffleDep.shuffleId, smt.partitionId, status) if (isChecksumMismatched) { diff --git a/core/src/main/scala/org/apache/spark/util/LastAttemptAccumulator.scala b/core/src/main/scala/org/apache/spark/util/LastAttemptAccumulator.scala new file mode 100644 index 0000000000000..cd17476a0cc40 --- /dev/null +++ b/core/src/main/scala/org/apache/spark/util/LastAttemptAccumulator.scala @@ -0,0 +1,867 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.util + +import scala.math.Ordering.Implicits._ +import scala.reflect.ClassTag +import scala.util.control.NonFatal + +import org.apache.spark.SparkContext +import org.apache.spark.internal.{LogEntry, Logging, LogKey, LogKeys} +import org.apache.spark.rdd.RDD +import org.apache.spark.scheduler.TaskInfo + +/* + * Last Attempt Accumulators are Accumulators that track the value of a metric aggregated across + * the "last execution" that produced the values. "Last execution" can be defined as: + * - For RDDs: the last execution of a given RDD partition, in the latest Stage and Stage attempt + * that recomputed it. + * - Across RDDs: lastAttemptValueForRDDId, lastAttemptValueForRDDIds, lastAttemptValueForAllRDDs, + * lastAttemptValueForHighestRDDId let specify that only values from specific RDDs should be + * aggregated. + * - For Spark SQL Execution: In SQLLastAttemptAccumulator, lastAttemptValueForDataset, + * lastAttemptValueForQueryExecution let specify that only values from the last SQL execution of + * a specific Dataset (or QueryExecution) should be aggregated. + * + * In specific situations the last attempt value cannot be computed. This is both because of known + * specific user actions (e.g. mixing driver updates with task updates), and because the + * accumulator performs (and logs) various internal sanity checks and bails out if it detects an + * unexpected situation. Therefore, all the lastAttempt methods return an Option[OUT], where None + * means that it has bailed out. + * + * Updates to the accumulator from completed Tasks are merged in mergeLastAttempt, called from + * DAGScheduler.updateAccumulators, called from DAGScheduler.handleTaskCompletion in the single + * threaded DAGScheduler event loop. Therefore, we don't need to worry about concurrency control + * when updating the accumulator values. However, reading of the last attempt value can potentially + * be done concurrently, so we use synchronization. When there is normally no contention, JVM + * synchronization should be very low overhead. + * + * In order to be able to provide last attempt value, we need to keep track of partial metric + * values, so that after a partial re-attempt the partial value can be updated, and then + * re-aggregated. + * There are various sources of re-attempts that we have to track: + * + * 1. Spark Core. + * ============== + * - Updates from failed tasks are filtered in Task.collectAccumulatorUpdates before they are + * even passed back to the driver. We don't need to worry about them here. + * - We should not get results from two successful attempts of a Task in the same Stage attempt. + * TaskSetManager.handleSuccessfulTask ensures that. + * - Therefore we only need to track Stage retries. The Last Attempt Metric will aggregate the + * metric value of a given RDD partition from the last attempt of the Stage with the highest + * stageId. + * Normally recomputation creates a new stageAttemptId in the same Stage, but there can also + * be multiple new Stages due to: + * - In AQE, a materialized QueryStage is submitted as a new Stage, which would normally get + * skipped, as it is already materialized. However, if results of that stage have been lost, + * the recomputation will happen in that Stage. + * - If the same Dataset with the same QueryExecution and same executedPlan is reused for + * another execution (e.g. again calling collect()). All map stages should be materialized, + * so like with AQE, they should be skipped, unless the results have been lost. Then, + * recomputation will happen in that Stage. The result stage computing the action will be + * fully re-executed. + * - Due to the async nature of cancellation, there can be tasks from previous attempts that + * arrive later than the last attempt. Therefore, we need to track and compare stageId and + * stageAttemptId of every computed RDD partition, in order to discard latecomers. + * + * 2. Spark SQL. + * ============= + * LastAttemptAccumulator offers simple tracking of the last SQL execution, by assuming that + * the last execution will be in the scope of an RDD with the highest id, and using + * [[lastAttemptValueForHighestRDDId]]. See SQLLastAttemptAccumulator for more possibilities + * of tracking SQL execution. + * + * Simple last SQL execution tracking + * ---------------------------------- + * Whenever an AQE replan happens, or a repeated execution is submitted, there will be a new + * RDD created for that execution. If AQE creates a new plan, it always uses it and cancels + * the previous one. So, aggregating the metric updates from the RDD with the highest id + * should correspond to the last execution and the latest AQE plan. + * This has some limitations, e.g. doesn't work if the same metric is used in multiple places + * in the query plan, and we want all occurrences to be aggregated together. + * It also wouldn't work if a SparkPlan splits its execution into multiple RDDs. This for example + * happens in BroadcastNestedLoopJoinExec with matchedStreamRows and notMatchedBroadcastRows. + * One can use this simple last attempt tracking by using lastAttemptLastRDDValue. + * + * 3. Driver only updates. + * ======================= + * Sometimes the metric is manipulated directly from the driver, not from within a Task. + * It can be either explicit by user code, or implicit by Catalyst Optimizer, for example + * ConvertToLocalRelation rule, folding a piece of the plan by evaluating it manually on the + * driver. + * When this happens, LastAttemptAccumulator has no information to reason about what was the + * last execution. If the only metric updates are coming from the driver, it assumes that these are + * the "last attempt". If there are both updates from executors and from the driver, it bails out. + * + * Implementation + * ============== + * To track the last attempts, we track a map of metric values per RDD id: + * - Map[RddId, LastAttemptRDDVals[PARTIAL]] + * + * In LastAttemptRDDVals we track an Array of per RDD partition partial merge values, together with + * the stageId and stageAttemptId and taskAttemptNumber to record task execution. + * We also track the RDD id, RDDScope id and last SQL execution id updating that RDD. + * + * Normally to merge partial values, two full Accumulators are used. However, accumulator classes + * that support Last Attempt have to implement partialMerge which merges PARTIAL type. + * This is used to have more compact representation, as PARTIAL can be e.g. a primitive type as + * opposed to a full AccumulatorV2 object instance. + */ + + +private class LastAttemptRDDVals[@specialized T]( + val rddId: Int, + val rddScopeId: Option[String], + // Arrays of partial metric values, and the corresponding stage, stage attempt and task attempt, + // with index representing RDD partition id. + // Metric updates to a given RDD partition can come from different stageAttempts if a retry + // happens while a Job with the Stage is running (a downstream Stage within a Job detects + // missing blocks and triggers recompute), or from different Stages, if a retry happens later + // (a new Job is submitted that depends on data from the RDD, if it finds it's missing it will + // recompute it in a new Stage). + // If a missing output is detected in a Stage while the stage is still running (e.g. executor + // is lost or decommissioned while the stage is running, and loses the output of some already + // finished tasks), a new Task with new taskAttemptNumber will be started for that Task. + // There may be multiple Tasks with different taskAttemptNumbers running in parallel due to + // speculation, but DAGScheduler guarantees that only one of them will reach metrics reporting, + // so it doesn't have to be dealt with here. + // + // There may be partitions that are either not computed at all (for example, due to early stop + // in take/limit), or AQE task coalescing may be visible as an update of the partition id of + // the first partition of the coalesced range. AQE guarantees that if these are retried, they + // will be coalesced in the same ranges, so update the same values. + // Not computed partitions should have EMPTY_ID in all the Int Arrays. + // + // Arrays of primitive types are more memory efficient than an array of objects due to + // references, object headers and paddings overheads. + // The `@specialized` annotation should make scala specialize it to use primitive array instead + // of boxed objects. + val partitionPartialVals: Array[T], + val stageIds: Array[Int], + val stageAttemptIds: Array[Int], + val taskAttemptNumbers: Array[Int]) + { + + // In a case of repeated execution of the same QueryExecution and reuse of the SparkPlan + // (for example multiple `collect()` on the same Dataset), a new RDD may be executed in the same + // RDDOperationScope for the new execution. Hence, we can have multiple RDDs with the same + // RDDOperationScope, coming from different SQL executions and we should only count the last one. + // However, it may also be an old RDD that is reused in the new execution, but needs to be + // partially recomputed because part of it is missing. In that case, the last attempt value needs + // to still be aggregated over the whole RDD, because the whole RDD is used in the new execution. + // Note that this only applies per RDDOperationScope/SparkPlan, because other plans in the same + // new execution may have reused their RDD in whole, and hence have the last SQL executionId + // come from an earlier execution. + // Note: This doesn't work in case a user concurrently executed multiple actions on the same + // Dataset, resulting in multiple concurrent executions trying to compute the same RDD. This + // however should not happen in practice and would likely produce other unexpected effects. + var lastSqlExecutionId: Option[Long] = None + + def numPartitions: Int = stageIds.length + + def isEmptyAt(partitionId: Int): Boolean = { + if (stageIds(partitionId) == LastAttemptRDDVals.EMPTY_ID) { + assert(stageAttemptIds(partitionId) == LastAttemptRDDVals.EMPTY_ID) + assert(taskAttemptNumbers(partitionId) == LastAttemptRDDVals.EMPTY_ID) + true + } else { + false + } + } + + def update(partialValue: AccumulatorPartialVal[T]): Unit = { + partitionPartialVals(partialValue.rddPartitionId) = partialValue.partialMergeVal + stageIds(partialValue.rddPartitionId) = partialValue.stageId + stageAttemptIds(partialValue.rddPartitionId) = partialValue.stageAttemptId + taskAttemptNumbers(partialValue.rddPartitionId) = partialValue.taskAttemptNumber + lastSqlExecutionId = partialValue.sqlExecutionId + } + + def partialValueAt(partId: Int): AccumulatorPartialVal[T] = { + AccumulatorPartialVal( + partialMergeVal = partitionPartialVals(partId), + rddId = rddId, + rddPartitionId = partId, + rddNumPartitions = stageIds.length, + rddScopeId = rddScopeId, + stageId = stageIds(partId), + stageAttemptId = stageAttemptIds(partId), + taskAttemptNumber = taskAttemptNumbers(partId), + sqlExecutionId = lastSqlExecutionId) + } + + override def toString: String = { + s"""LastAttemptVal( + | rddId=$rddId, + | rddScopeId=$rddScopeId, + | lastSqlExecutionId=$lastSqlExecutionId, + | partitionPartialVals=${partitionPartialVals.mkString("[", ",", "]")}, + | stageIds=${stageIds.mkString("[", ",", "]")}, + | stageAttemptIds=${stageAttemptIds.mkString("[", ",", "]")}, + | taskAttemptNumbers=${taskAttemptNumbers.mkString("[", ",", "]")} + |)""".stripMargin + } +} + +private object LastAttemptRDDVals { + // EMPTY_ID in stageId means that the partition was not computed. + val EMPTY_ID: Int = -1 + + def apply[@specialized T]( + rddId: Int, + rddScopeId: Option[String], + numPartitions: Int)(implicit ct: ClassTag[T]): LastAttemptRDDVals[T] = { + new LastAttemptRDDVals[T]( + rddId, + rddScopeId, + new Array[T](numPartitions), + Array.fill(numPartitions)(LastAttemptRDDVals.EMPTY_ID), + Array.fill(numPartitions)(LastAttemptRDDVals.EMPTY_ID), + Array.fill(numPartitions)(LastAttemptRDDVals.EMPTY_ID)) + } + + def createFromFirstUpdate[@specialized T]( + update: AccumulatorPartialVal[T])(implicit ct: ClassTag[T]): LastAttemptRDDVals[T] = { + val newVal = LastAttemptRDDVals[T]( + rddId = update.rddId, + rddScopeId = update.rddScopeId, + update.rddNumPartitions) + newVal.update(update) + newVal + } +} + +private class LastAttemptMap[K, V] { + // Map used to keep metric updates, keyed by RDD id or RDD scope id, backed by a List. + // In the majority of cases (when there are no stage retries and no AQE replanning + // cancelling already running stages), there will be only one key, so a list backed map + // should have less overhead. + // + // Accumulators are modified only from DAGScheduler.updateAccumulators -> mergeLastAttempt, + // which is running from a single thread (scheduling loop), so no concurrency control is needed + // for updates. Read accesses to an immutable list should use a consistent state without extra + // synchronization. + + @volatile private var map: List[(K, V)] = Nil + + def contains(key: K): Boolean = map.exists(_._1 == key) + + def get(key: K): Option[V] = map.collectFirst { case (k, v) if k == key => v } + + def put(key: K, value: V): Unit = synchronized { + map = (key, value) :: map.filterNot(_._1 == key) + } + + def keys: Iterable[K] = map.map(_._1) + def values: Iterable[V] = map.map(_._2) + def isEmpty: Boolean = map.isEmpty + def nonEmpty: Boolean = map.nonEmpty + def clear(): Unit = synchronized { map = Nil } + + override def toString: String = map + .map(elem => s"${elem._1} -> ${elem._2}").mkString("LastAttemptMap {\n", ",\n", "\n}") +} + +private case class AccumulatorPartialVal[PARTIAL]( + partialMergeVal: PARTIAL, + rddId: Int, + rddPartitionId: Int, + rddNumPartitions: Int, + rddScopeId: Option[String], + stageId: Int, + stageAttemptId: Int, + taskAttemptNumber: Int, + sqlExecutionId: Option[Long] +) { + override def toString: String = { + s"""AccumulatorPartialVal( + | partialMergeVal=$partialMergeVal, + | rddId=$rddId, + | rddPartitionId=$rddPartitionId, + | rddNumPartitions=$rddNumPartitions, + | rddScopeId=$rddScopeId, + | stageId=$stageId, + | stageAttemptId=$stageAttemptId, + | taskAttemptNumber=$taskAttemptNumber, + | sqlExecutionId=$sqlExecutionId + |)""".stripMargin + } + + /** Tuple of stage id, stage attempt id and taskAttemptNumber, defining the order of attempts. */ + val attempt: (Int, Int, Int) = (stageId, stageAttemptId, taskAttemptNumber) +} + +/** + * A trait that can be mixed into a subclass of [[AccumulatorV2]] to track the "logical" + * value of the "last attempt" of the execution using the accumulator - aggregated from the last + * attempts of any Task that calculated some RDD partitions and used this accumulator, and + * discarding any values coming from earlier attempts that have been recomputed. + * If the accumulator is used by multiple RDDs, the last attempt value is tracked separately for + * each, and can be retrieved for each or all of them separately, see lastAttemptValueForX methods. + * If the accumulator is used directly on the Spark Driver using [[AccumulatorV2#add]], + * that value is considered the last attempt value. + * If the accumulator was both used in Tasks and updated directly on the driver, it can't determine + * what should be considered the last attempt, and lastAttemptValueForX methods will return None. + * + * Contract for driver-only updates: + * A driver-side value (set via [[AccumulatorV2#add]] on the driver, outside any Task) is only + * returned by methods that do not narrow by RDD, namely [[lastAttemptValueForAllRDDs]] and + * [[lastAttemptValueForHighestRDDId]]. Methods that narrow to specific RDDs or RDD scopes + * ([[lastAttemptValueForRDDId]], [[lastAttemptValueForRDDIds]], [[lastAttemptValueForRDDScopes]]) + * return the zero value when a driver-only value is present, because a driver-side update cannot + * be attributed to any particular RDD or scope. + * + * [[LastAttemptAccumulator]] is not reset by the [[AccumulatorV2#reset]] method implementation, + * and its state is not copied by the [[AccumulatorV2#copy]] method implementation, and it should + * not be serialized to the Executors. The internal state should only be initialized by the + * [[initializeLastAttemptAccumulator]] method on the "main" instance of the accumulator, that was + * created and registered with [[AccumulatorContext]] with [[AccumulatorV2#register]]. All the + * interfaces of [[LastAttemptAccumulator]]: [[mergeLastAttempt]] (used only by DAGScheduler) and + * lastAttemptValueForX, [[logAccumulatorState]] (used by the using code) should only be invoked on + * that instance, on the Spark Driver. + * + * The [[LastAttemptAccumulator]] is not thread-safe. [[mergeLastAttempt]] should only be used by + * DAGScheduler, by the scheduler thread. Retrieving the value using lastAttemptValueForXXX while + * it is concurrently updated (execution is running) can produce some inconsistencies, but should + * not crash. + * If an RDD using the [[LastAttemptAccumulator]] is used concurrently by multiple actions that + * all try to recompute it, it may produce unexpected results and the semantics of what is "last + * attempt" becomes ambiguous. This should not be done in practice, and will likely result in more + * unexpected behaviours in Spark. + * + * Implementations must implement [[partialMergeVal]] and [[partialMerge]] methods operating on + * PARTIAL type. In regular [[AccumulatorV2]] implementations, the [[AccumulatorV2]] object + * itself holds the intermediate value of the accumulator, and [[AccumulatorV2#merge]] method is + * used to merge these objects together. [[LastAttemptAccumulator]] needs to keep track of partial + * values of every partition of every RDD that used the accumulator, and holding a full + * [[AccumulatorV2]] object for each would have a high overhead. Therefore, an implementation should + * be able to return PARTIAL value from [[partialMergeVal]] that represents an intermediate + * mergeable value, and a [[partialMerge]] method that can merge that value into the accumulator. + * Implementations must also implement an [[isMergeable]] method that checks if the other + * [[AccumulatorV2]] is of a compatible type to be merged with this using [[partialMergeVal]]. In + * regular [[AccumulatorV2]] implementations, this check is normally done inside the + * [[AccumulatorV2#merge]] method, which is not used here. + * + * If an implementation is used to keep user data in the accumulator, it should override + * [[accumulatorStoresUserData]] to return true, to ensure correct structured logging annotation. + * Otherwise it should override it to false. + */ +trait LastAttemptAccumulator[IN, OUT, PARTIAL] extends Logging { + this: AccumulatorV2[IN, OUT] => + + // For every RDD that participated in the computation of this accumulator, keep the partial + // value of the accumulator for the latest stage and stage attempt that computed it. + // Keyed by rdd.id. + // Only kept and accessed on the driver, in the instance of the LastAttemptAccumulator that was + // created and registered with AccumulatorContext with AccumulatorV2.register(). + // Should not be copied / reset by the implementation of copy() / reset() functions. + // Transient: only needed on the driver and doesn't need to be serialized. + @transient + private var lastAttemptRddsMap: LastAttemptMap[Int, LastAttemptRDDVals[PARTIAL]] = _ + + // ClassTag for PARTIAL, captured at initialization time. + @transient private var partialClassTag: ClassTag[PARTIAL] = _ + + // Metric value set directly on the driver, not from within a task. + // Only kept and accessed on the driver, in the instance of the LastAttemptAccumulator that was + // created and registered with AccumulatorContext with AccumulatorV2.register(). + // Should not be copied / reset by the implementation of copy() / reset() functions. + // Transient: only needed on the driver and doesn't need to be serialized. + @transient + private var lastAttemptDirectDriverValue: Option[OUT] = _ + + // Flipped to true if unexpected metrics updates are received and we can no longer reason + // about the last attempt. + // Should not be copied / reset by the implementation of copy() / reset() functions. + // Transient: only needed on the driver and doesn't need to be serialized. + @transient + protected var lastAttemptAccumulatorInvalid: Boolean = false + + // Indicates that the LastAttemptAccumulator has been initialized. + // It is initialized in assertValid(). + // Should not be copied / reset by the implementation of copy() / reset() functions. + // Transient: only needed on the driver and doesn't need to be serialized. + @transient + protected var lastAttemptAccumulatorInitialized: Boolean = false + + /** Reset the state of the last attempt accumulator, discarding all the past attempts, and + * making it valid again if it was invalidated. */ + def resetLastAttemptAccumulator(): Unit = try { + lastAttemptRddsMap.clear() + lastAttemptDirectDriverValue = None + lastAttemptAccumulatorInvalid = false + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in resetLastAttemptAccumulator", + exception = Some(e)) + } + + def initializeLastAttemptAccumulator()(implicit ct: ClassTag[PARTIAL]): Unit = try { + assert(isAtDriverSide) + assert(!lastAttemptAccumulatorInitialized) + assert(!lastAttemptAccumulatorInvalid) + assert(lastAttemptRddsMap == null) + assert(lastAttemptDirectDriverValue == null) + partialClassTag = ct + lastAttemptRddsMap = new LastAttemptMap[Int, LastAttemptRDDVals[PARTIAL]] + lastAttemptDirectDriverValue = None + lastAttemptAccumulatorInitialized = true + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in initializeLastAttemptAccumulator", + exception = Some(e)) + } + + private def accumulatorId: Long = { + // This can throw if this is a copy/serialized accumulator, + // not the instance registered with AccumulatorContext. + // Catch it so we can safely use it for logging in unexpected situations. + try { + this.id + } catch { + case NonFatal(e) => + logWarning(log"Unexpected exception in getting accumulator id", e) + -1L // needs to be a long for LogKeys.ACCUMULATOR_ID + } + } + + /** Log entry to log debug information about the internal state of the accumulator. */ + def logAccumulatorState: LogEntry = try { + log"""LastAttemptAccumulator id=${MDC(LogKeys.ACCUMULATOR_ID, accumulatorId)}: + |Invalidated: ${MDC(LogKeys.LAST_ATTEMPT_ACC_INVALIDATE, lastAttemptAccumulatorInvalid)}. + |Direct driver value: ${MDC(logKeyAccumulatorState, lastAttemptDirectDriverValue)}. + |Value: ${MDC(logKeyAccumulatorState, value)}. + |lastAttemptRddsMap: + |${MDC(logKeyAccumulatorState, lastAttemptRddsMap)}.""" + .stripMargin + } catch { + case NonFatal(e) => + logWarning(log"Unexpected exception in logAccumulatorState", e) + log"" + } + + private def logAccumulatorUpdate( + newAccumPartialValue: Option[AccumulatorPartialVal[PARTIAL]] = None, + oldAccumPartialValue: Option[AccumulatorPartialVal[PARTIAL]] = None): LogEntry = try { + log"""Old partial RDD value: ${MDC(logKeyAccumulatorState, oldAccumPartialValue)}. + |New partial RDD value: ${MDC(logKeyAccumulatorState, newAccumPartialValue)}.""" + .stripMargin + } catch { + case NonFatal(e) => + logWarning(log"Unexpected exception in logAccumulatorUpdate", e) + log"" + } + + private def unexpectedLastAttemptMetricUpdate( + invalidate: Boolean, + reason: String, + exception: Option[Throwable] = None, + newAccumPartialValue: Option[AccumulatorPartialVal[PARTIAL]] = None, + oldAccumPartialValue: Option[AccumulatorPartialVal[PARTIAL]] = None): Unit = { + val logEntry = + log"""Unexpected last attempt tracking for accumulator ${ + MDC(LogKeys.ACCUMULATOR_ID, accumulatorId)}. + |Invalidate: ${MDC(LogKeys.LAST_ATTEMPT_ACC_INVALIDATE, invalidate)}. + |Reason: ${MDC(LogKeys.LAST_ATTEMPT_ACC_UNEXPECTED_REASON, reason)}. + |""".stripMargin + + log"State:\n" + logAccumulatorState + + log"Update:\n" + logAccumulatorUpdate(newAccumPartialValue, oldAccumPartialValue) + exception match { + case Some(e) => logWarning(logEntry, e) + case None => logWarning(logEntry) + } + if (invalidate) { + lastAttemptAccumulatorInvalid = true + } + if (Utils.isTesting && lastAttemptAccumulatorInitialized && exception.isDefined) { + // If this is a test, rethrow the exception. + // (Rethrow only if lastAttemptAccumulatorInitialized. In some tests, we check for proper + // graceful handling of unexpected exceptions in accumulators that are not properly + // initialized, so we don't want to throw there.) + throw exception.get + } + } + + protected def unexpectedLastAttemptMetricOperation( + invalidate: Boolean, + reason: String, + exception: Option[Throwable] = None): Unit = { + // subclasses don't have visibility of private class AccumulatorPartialVal. + unexpectedLastAttemptMetricUpdate( + invalidate = invalidate, + reason = reason, + exception = exception, + newAccumPartialValue = None, + oldAccumPartialValue = None) + } + + /** Set of assertions that should always hold for a valid [[LastAttemptAccumulator]]. */ + protected def assertValid(): Unit = { + assert(lastAttemptAccumulatorInitialized) + assert(!lastAttemptAccumulatorInvalid) + assert(isAtDriverSide) + assert(metadata != null) + assert(!metadata.countFailedValues) + assert(lastAttemptDirectDriverValue.isEmpty || lastAttemptRddsMap.isEmpty) + } + + /** + * Accumulator subclasses where metric values can contain user data (for example, maximum of + * processed values, observable metrics) as opposed to system measurements (for example, count + * of processed rows) should return true to ensure correct structured logging annotation. + */ + protected def accumulatorStoresUserData: Boolean + + protected def logKeyAccumulatorState: LogKey = { + if (accumulatorStoresUserData) { + LogKeys.LAST_ATTEMPT_ACC_USER_METRIC + } else { + LogKeys.LAST_ATTEMPT_ACC_SYSTEM_METRIC + } + } + + /** Return intermediate value of PARTIAL type that can be merged together by partialMerge. */ + protected def partialMergeVal: PARTIAL + + /** Merge together partial values of PARTIAL type returned by partialMergeVal. */ + protected def partialMerge(otherVal: PARTIAL): Unit + + /** Check if the other accumulator is mergeable with this one. */ + protected def isMergeable(other: AccumulatorV2[_, _]): Boolean + + /** + * Check if the value is set on the driver side, not from within a task. + * This must be called from `add` and `set` methods of any AccumulatorV2 subclass supporting + * last attempt metrics to set what the `value` of the metric is after the operation. + */ + protected def setValueIfOnDriverSide(value: OUT): Unit = try { + if (isAtDriverSide && lastAttemptAccumulatorInitialized && !lastAttemptAccumulatorInvalid) { + // Direct update on the driver, not from within a task. + // This gives little information about the source of the update, so we can't reason about + // "last attempt" if it's mixed with non-driver updates. + lastAttemptDirectDriverValue = Some(value) + if (lastAttemptRddsMap.nonEmpty) { + unexpectedLastAttemptMetricUpdate( + invalidate = true, + reason = "Incoming direct driver value while task updates exist") + } + } + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in setValueIfOnDriverSide", + exception = Some(e)) + } + + /** + * It needs Task and Stage information to reason about the last attempt. + * + * Called from a single thread in DAGScheduler, no synchronization needed. + * Should be used only on the Spark Driver, on the instance of [[LastAttemptAccumulator]] that + * was created and registered in [[AccumulatorContext]] by [[AccumulatorV2#register]]. + */ + private[spark] def mergeLastAttempt( + other: AccumulatorV2[_, _], + rdd: RDD[_], + taskInfo: TaskInfo, + stageId: Int, + stageAttemptId: Int, + localProperties: java.util.Properties): Unit = try { + implicit val ct: ClassTag[PARTIAL] = partialClassTag + if (lastAttemptAccumulatorInvalid) return + // Skip zero-value updates. They contribute nothing to the aggregate and can come + // from stages where the accumulator was present in the task closure but never incremented. + if (other.isZero) return + assertValid() + + if (!isMergeable(other)) { + // This should never happen. + unexpectedLastAttemptMetricUpdate( + invalidate = true, + "Merging accumulators of different types") + return + } + + if (!other.isInstanceOf[LastAttemptAccumulator[_, _, _]]) { + // This should never happen. + unexpectedLastAttemptMetricUpdate( + invalidate = true, + "Merging with accumulator which is not SLAM") + return + } + val lastAttemptOther = other + .asInstanceOf[LastAttemptAccumulator[IN, OUT, PARTIAL]] + + val update = AccumulatorPartialVal( + partialMergeVal = lastAttemptOther.partialMergeVal, + rddId = rdd.id, + rddPartitionId = taskInfo.partitionId, + rddNumPartitions = rdd.getNumPartitions, + rddScopeId = rdd.scope.map(_.id), + stageId = stageId, + stageAttemptId = stageAttemptId, + taskAttemptNumber = taskInfo.attemptNumber, + sqlExecutionId = + Option(localProperties.getProperty(SparkContext.SQL_EXECUTION_ID_KEY)).map(_.toLong)) + + if (lastAttemptDirectDriverValue.nonEmpty) { + unexpectedLastAttemptMetricUpdate(invalidate = true, + "Incoming task updates while direct driver value exists", + newAccumPartialValue = Some(update)) + return + } + + lastAttemptRddsMap.get(update.rddId) match { + case Some(oldRDDValue) => // This RDD was already seen. + val oldValue = oldRDDValue.partialValueAt(update.rddPartitionId) + + logTrace(log"mergeLastAttempt existing RDD update:\n" + + log"${MDC(logKeyAccumulatorState, oldRDDValue)}\n" + + logAccumulatorUpdate( + newAccumPartialValue = Some(update), oldAccumPartialValue = Some(oldValue))) + + // Check basic consistency + if (oldValue.rddNumPartitions != update.rddNumPartitions) { + unexpectedLastAttemptMetricUpdate( + invalidate = true, + reason = "RDD with changing number of partitions", + newAccumPartialValue = Some(update), + oldAccumPartialValue = Some(oldValue)) + return + } + if (oldValue.rddScopeId != update.rddScopeId) { + unexpectedLastAttemptMetricUpdate( + invalidate = true, + reason = "RDD with changing RDDOperationScope", + newAccumPartialValue = Some(update), + oldAccumPartialValue = Some(oldValue)) + return + } + + if (oldRDDValue.isEmptyAt(update.rddPartitionId)) { + // No previous attempt for this RDD partition. + oldRDDValue.update(update) + } else { + if (update.attempt > oldValue.attempt) { + // New last attempt for this RDD partition. + oldRDDValue.update(update) + } else if (update.attempt == oldValue.attempt) { + // Same attempt, should not happen. + unexpectedLastAttemptMetricUpdate( + invalidate = true, + reason = "Same stage, stageAttemptId and taskAttemptNumber reported multiple times", + newAccumPartialValue = Some(update), + oldAccumPartialValue = Some(oldValue)) + } + // else: Older attempt reported after newer attempt. Not fatal, discard it. + } + + case None => // First time we see this RDD. + logTrace(log"mergeLastAttempt new RDD update:\n" + logAccumulatorUpdate( + newAccumPartialValue = Some(update), oldAccumPartialValue = None)) + val newVal = LastAttemptRDDVals.createFromFirstUpdate(update) + lastAttemptRddsMap.put(update.rddId, newVal) + } + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricUpdate( + invalidate = true, + reason = "Unexpected exception in mergeLastAttempt", + exception = Some(e)) + } + + /** Accumulates last attempt values from given RDD into an acc. */ + private def lastAttemptValueAggregateInternal(rddId: Int, acc: this.type) = { + // Note: even if the given RDD is not present, we can't tell if it executed but just never + // updated this accumulator, so we still report the zero value back. + for { + lastAttemptVal <- lastAttemptRddsMap.get(rddId) + partitionId <- lastAttemptVal.partitionPartialVals.indices + } { + // Some partitions may not be computed. + // May be because of operations like take. + // May be because of AQE coalescing executing tasks covering multiple partitions. + if (!lastAttemptVal.isEmptyAt(partitionId)) { + acc.partialMerge(lastAttemptVal.partitionPartialVals(partitionId)) + } + } + } + + /** + * Returns the last attempt value of this accumulator, aggregated from a set of RDDs. + * + * Should be used only on the Spark Driver, on the instance of [[LastAttemptAccumulator]] that + * was created and registered in [[AccumulatorContext]] by [[AccumulatorV2#register]]. + * + * @return None if the last attempt value cannot be established, Some(value) otherwise. + */ + def lastAttemptValueForRDDIds(rddIds: Seq[Int]): Option[OUT] = try { + if (lastAttemptAccumulatorInvalid) return None + assertValid() + if (lastAttemptDirectDriverValue.isDefined) { + // return zero value if there is no RDD execution recorded. + return Some(copyAndReset().asInstanceOf[this.type].value) + } + + val acc = copyAndReset().asInstanceOf[this.type] + rddIds.distinct.foreach(lastAttemptValueAggregateInternal(_, acc)) + Some(acc.value) + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in lastAttemptValueForRDDs", + exception = Some(e)) + None + } + + /** + * Returns the last attempt value of this accumulator, aggregated from a specific RDD. + * + * Should be used only on the Spark Driver, on the instance of [[LastAttemptAccumulator]] that + * was created and registered in [[AccumulatorContext]] by [[AccumulatorV2#register]]. + * + * @return None if the last attempt value cannot be established, Some(value) otherwise. + */ + def lastAttemptValueForRDDId(rddId: Int): Option[OUT] = try { + lastAttemptValueForRDDIds(Seq(rddId)) + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in lastAttemptValueForRDD", + exception = Some(e)) + None + } + + /** + * Returns the last attempt value of this accumulator, aggregated from all RDDs that ever + * returned any values for it. + * + * If the metric was used directly on the driver, and was not used in any RDD execution, + * the driver value will be used instead. + * + * Should be used only on the Spark Driver, on the instance of [[LastAttemptAccumulator]] that + * was created and registered in [[AccumulatorContext]] by [[AccumulatorV2#register]]. + * + * @return None if the last attempt value cannot be established, Some(value) otherwise. + */ + def lastAttemptValueForAllRDDs(): Option[OUT] = try { + if (lastAttemptAccumulatorInvalid) return None + assertValid() + if (lastAttemptDirectDriverValue.isDefined) return lastAttemptDirectDriverValue + lastAttemptValueForRDDIds(lastAttemptRddsMap.keys.toSeq) + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in lastAttemptValueForAllRDDs", + exception = Some(e)) + None + } + + /** + * Returns the last attempt value of this accumulator, aggregated from the RDD with the highest + * id that ever returned any values for it. + * + * If the metric was used directly on the driver, and was not used in any RDD execution, + * the driver value will be used instead. + * + * Should be used only on the Spark Driver, on the instance of [[LastAttemptAccumulator]] that + * was created and registered in [[AccumulatorContext]] by [[AccumulatorV2#register]]. + * + * @return None if the last attempt value cannot be established, Some(value) otherwise. + */ + def lastAttemptValueForHighestRDDId(): Option[OUT] = try { + if (lastAttemptAccumulatorInvalid) return None + assertValid() + if (lastAttemptDirectDriverValue.isDefined) return lastAttemptDirectDriverValue + + if (lastAttemptRddsMap.nonEmpty) { + lastAttemptValueForRDDId(lastAttemptRddsMap.keys.max) + } else { + // return zero value if there is no RDD execution recorded. + Some(copyAndReset().asInstanceOf[this.type].value) + } + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in lastAttemptValueForHighestRDDId", + exception = Some(e)) + None + } + + /** + * Returns the last attempt value of this accumulator, aggregated from RDDs with given scope ids. + * + * Should be used only on the Spark Driver, on the instance of [[LastAttemptAccumulator]] that + * was created and registered in [[AccumulatorContext]] by [[AccumulatorV2#register]]. + * + * @return None if the last attempt value cannot be established, Some(value) otherwise. + */ + def lastAttemptValueForRDDScopes(rddScopeIds: Seq[String]): Option[OUT] = try { + if (lastAttemptAccumulatorInvalid) return None + assertValid() + if (lastAttemptDirectDriverValue.isDefined) { + // Return zero value if there is no RDD execution recorded. + return Some(copyAndReset().asInstanceOf[this.type].value) + } + val scopesLookup = rddScopeIds.toSet + val matchingRDDs = lastAttemptRddsMap.values.filter { rddVal => + rddVal.rddScopeId.exists(scopesLookup.contains) + }.toSeq + // When multiple RDDs share the same scope (e.g. repeated Dataset.collect() calls create + // new wrapper RDDs in the same scope, or BroadcastNestedLoopJoin executing the probe side + // twice), only aggregate the latest one per scope, identified by the highest RDD id. + // RDD ids are globally monotonic, so the highest id is the latest. + val rddIds = matchingRDDs.groupBy(_.rddScopeId).values.map(_.maxBy(_.rddId).rddId).toSeq + lastAttemptValueForRDDIds(rddIds) + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in lastAttemptValueForRDDScopes", + exception = Some(e)) + None + } + + /** Visible for testing. */ + def getDirectDriverValue: Option[OUT] = { + lastAttemptDirectDriverValue + } + + /** Visible for testing */ + def getHighestRDDId: Option[Int] = { + if (lastAttemptRddsMap.nonEmpty) Some(lastAttemptRddsMap.keys.max) else None + } + + /** Visible for testing */ + def getNumRDDs: Int = { + lastAttemptRddsMap.keys.size + } + + /** Visible for testing */ + def getValid: Boolean = { + !lastAttemptAccumulatorInvalid + } +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala b/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala index 5bef4e35ba57e..91d51163b319e 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala @@ -2259,9 +2259,11 @@ class Dataset[T] private[sql]( */ private def withAction[U](name: String, qe: QueryExecution)(action: SparkPlan => U) = { SQLExecution.withNewExecutionId(qe, Some(name)) { - QueryExecution.withInternalError(s"""The "$name" action failed.""") { - qe.executedPlan.resetMetrics() - action(qe.executedPlan) + qe.withQueryExecutionId(sparkSession) { + QueryExecution.withInternalError(s"""The "$name" action failed.""") { + qe.executedPlan.resetMetrics() + action(qe.executedPlan) + } } } } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala index f08b561d6ef9a..c0ab906de4841 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala @@ -26,7 +26,7 @@ import scala.util.control.NonFatal import org.apache.hadoop.fs.Path -import org.apache.spark.SparkException +import org.apache.spark.{SparkContext, SparkException} import org.apache.spark.internal.Logging import org.apache.spark.internal.LogKeys.EXTENDED_EXPLAIN_GENERATOR import org.apache.spark.rdd.RDD @@ -327,7 +327,30 @@ class QueryExecution( protected def executePhase[T](phase: String)(block: => T): T = sparkSession.withActive { QueryExecution.withInternalError(s"The Spark SQL phase $phase failed with an internal error.") { - tracker.measurePhase(phase)(block) + withQueryExecutionId(sparkSession) { + tracker.measurePhase(phase)(block) + } + } + } + + /** + * Set the query execution id in thread-local properties while + * executing the block. This is used by + * [[org.apache.spark.sql.execution.metric.SQLLastAttemptAccumulator]] to associate + * driver-side metric updates with a specific QueryExecution. + */ + private[sql] def withQueryExecutionId[T]( + session: SparkSession)(block: => T): T = { + val sc = session.sparkContext + val oldId = sc.getLocalProperty( + SparkContext.DATASET_QUERY_EXECUTION_ID_KEY) + sc.setLocalProperty( + SparkContext.DATASET_QUERY_EXECUTION_ID_KEY, id.toString) + try { + block + } finally { + sc.setLocalProperty( + SparkContext.DATASET_QUERY_EXECUTION_ID_KEY, oldId) } } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala index 7f94cc77f3454..cf2d0218d0fdd 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala @@ -249,12 +249,21 @@ abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializ doExecuteWrite(writeFilesSpec) } + /** + * A deterministic scope ID for RDDs created by this SparkPlan, + * used by LastAttemptAccumulator to track which RDD belongs + * to which SparkPlan node. + */ + private[spark] def rddScopeId: String = + "spark_plan_" + id.toString + /** * Executes a query after preparing the query and adding query plan information to created RDDs * for visualization. */ protected final def executeQuery[T](query: => T): T = { - RDDOperationScope.withScope(sparkContext, nodeName, false, true) { + RDDOperationScope.withScope( + sparkContext, nodeName, false, true, rddScopeId) { prepare() waitForSubqueries() query @@ -375,6 +384,11 @@ abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializ */ private def getByteArrayRdd( n: Int = -1, takeFromEnd: Boolean = false): RDD[(Long, ChunkedByteBuffer)] = { + // Wrap in the plan's RDD scope so that the wrapper RDD created by mapPartitionsInternal + // inherits this plan's deterministic scope ID rather than getting an anonymous auto-generated + // one. + val rdd = RDDOperationScope.withScope( + sparkContext, nodeName, false, true, rddScopeId) { execute().mapPartitionsInternal { iter => var count = 0 val buffer = new Array[Byte](4 << 10) // 4K @@ -409,8 +423,10 @@ abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializ out.writeInt(-1) out.flush() out.close() - Iterator((count, cbbos.toChunkedByteBuffer)) + Iterator((count.toLong, cbbos.toChunkedByteBuffer)) + } } + rdd } /** diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQETestHelper.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQETestHelper.scala new file mode 100644 index 0000000000000..7e78ee41900ce --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQETestHelper.scala @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.execution.adaptive + +import scala.collection.mutable + +import org.apache.spark.sql.execution.SparkPlan +import org.apache.spark.sql.execution.metric.SQLMetric +import org.apache.spark.util.{AccumulatorContext, Utils} + +/** Testing only helpers for AQE. */ +object AQETestHelper { + // See [withForcedCancellation]. + @volatile private var metricIdsForForcedCancellation: Set[Long] = Set.empty + + /** + * Set `triggerMetrics` to induce a forced cancellation into the execution when any of the + * metrics is non-empty. + * In this case the results will be discarded and the stage re-run, causing the metrics to be + * incremented again. + */ + def withForcedCancellation[T](triggerMetrics: SQLMetric*)(thunk: => T): T = { + metricIdsForForcedCancellation = triggerMetrics.map(_.id).toSet + val res = try { + thunk + } finally { + metricIdsForForcedCancellation = Set.empty + forcedCancellationTriggeredForPlans.clear() + } + res + } + + /* + * Track for which plans we have already triggered the forced replanning so we only do it once. + */ + private val forcedCancellationTriggeredForPlans = mutable.HashSet.empty[Int] + + /** Return `true` if forced cancellation mechanism is enabled. */ + def isForcedCancellationEnabled: Boolean = + Utils.isTesting && metricIdsForForcedCancellation.nonEmpty + + /** Return `true` if forced cancellation has already been triggered for `plan`. */ + private def wasForcedCancellationTriggeredForPlan(plan: SparkPlan): Boolean = synchronized { + forcedCancellationTriggeredForPlans.contains(plan.id) + } + + /** Mark that force cancellation was successfully triggered for `plan`. */ + def markForcedCancellationTriggeredForPlan(plan: SparkPlan): Unit = synchronized { + assert(!forcedCancellationTriggeredForPlans.contains(plan.id), + "A plan was forced to cancel a second time.") + forcedCancellationTriggeredForPlans += plan.id + } + + /** Return `true` if we should try to force cancellation for `plan` at this point. */ + def shouldForceCancellation(plan: SparkPlan): Boolean = { + // Trigger the forced cancellation only if we are in testing + Utils.isTesting && + // ...and if we haven't triggered it yet + !wasForcedCancellationTriggeredForPlan(plan) && + // ...and if any of the trigger metrics > 0. + metricIdsForForcedCancellation.exists { id => + AccumulatorContext.get(id).map(!_.isZero).getOrElse(false) + } + } +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala index 4840016bf745d..112ee82314c4b 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala @@ -340,6 +340,8 @@ case class AdaptiveSparkPlanExec( if (errors.nonEmpty) { cleanUpAndThrowException(errors.toSeq, None) } + val testTriggerForceCancellation = AQETestHelper.shouldForceCancellation(this) + if (!currentPhysicalPlan.isInstanceOf[ResultQueryStageExec]) { // Try re-optimizing and re-planning. Adopt the new plan if its cost is equal to or less // than that of the current plan; otherwise keep the current physical plan together with @@ -352,14 +354,21 @@ case class AdaptiveSparkPlanExec( // the current physical plan. Once a new plan is adopted and both logical and physical // plans are updated, we can clear the query stage list because at this point the two // plans are semantically and physically in sync again. - val logicalPlan = replaceWithQueryStagesInLogicalPlan(currentLogicalPlan, stagesToReplace) + var logicalPlan = replaceWithQueryStagesInLogicalPlan(currentLogicalPlan, stagesToReplace) + if (testTriggerForceCancellation) { + // Force unwrap all LogicalQueryStage so they get replanned. + logicalPlan = logicalPlan.transformDown { + case LogicalQueryStage(logical, _) => logical + } + } val afterReOptimize = reOptimize(logicalPlan) if (afterReOptimize.isDefined) { val (newPhysicalPlan, newLogicalPlan) = afterReOptimize.get val origCost = costEvaluator.evaluateCost(currentPhysicalPlan) val newCost = costEvaluator.evaluateCost(newPhysicalPlan) if (newCost < origCost || - (newCost == origCost && currentPhysicalPlan != newPhysicalPlan)) { + (newCost == origCost && currentPhysicalPlan != newPhysicalPlan) || + testTriggerForceCancellation) { lazy val plans = sideBySide( currentPhysicalPlan.treeString, newPhysicalPlan.treeString).mkString("\n") logOnLevel(log"Plan changed:\n${MDC(QUERY_PLAN, plans)}") @@ -369,6 +378,9 @@ case class AdaptiveSparkPlanExec( stagesToReplace = Seq.empty[QueryStageExec] } } + if (testTriggerForceCancellation) { + AQETestHelper.markForcedCancellationTriggeredForPlan(this) + } } // Now that some stages have finished, we can try creating new stages. result = createQueryStages(fun, currentPhysicalPlan, firstRun = false) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanHelper.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanHelper.scala index 2556edee8d02f..eea664b29fd52 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanHelper.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanHelper.scala @@ -94,6 +94,21 @@ trait AdaptiveSparkPlanHelper { collect(p) { case plan if allChildren(plan).isEmpty => plan } } + /** + * Returns true if the condition specified by `f` is satisfied by any node in this tree. + */ + def exists(p: SparkPlan)(f: SparkPlan => Boolean): Boolean = { + find(p)(f).isDefined + } + + /** + * Like [[exists]], but also considers plan nodes inside subqueries. + */ + def existsWithSubqueries( + p: SparkPlan)(f: SparkPlan => Boolean): Boolean = { + exists(p)(f) || subqueriesAll(p).exists(exists(_)(f)) + } + /** * Finds and returns the first [[SparkPlan]] of the tree for which the given partial function * is defined (pre-order), and applies the partial function to it. @@ -138,3 +153,5 @@ trait AdaptiveSparkPlanHelper { case other => other } } + +private[sql] object AdaptiveSparkPlanHelper extends AdaptiveSparkPlanHelper diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala index 7dcbf3779b93d..7f757c651c560 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala @@ -241,17 +241,22 @@ case class ShuffleExchangeExec( */ @transient lazy val shuffleDependency : ShuffleDependency[Int, InternalRow, InternalRow] = { - val dep = ShuffleExchangeExec.prepareShuffleDependency( - inputRDD, - child.output, - outputPartitioning, - serializer, - writeMetrics) - metrics("numPartitions").set(dep.partitioner.numPartitions) - val executionId = sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY) - SQLMetrics.postDriverMetricUpdates( - sparkContext, executionId, metrics("numPartitions") :: Nil) - dep + // Wrap in the exchange's RDD scope so that any wrapper RDDs created during shuffle dependency + // preparation (e.g. by prepareShuffleDependency's mapPartitionsInternal calls) get this + // exchange's scope ID. + RDDOperationScope.withScope(sparkContext, nodeName, false, true, rddScopeId) { + val dep = ShuffleExchangeExec.prepareShuffleDependency( + inputRDD, + child.output, + outputPartitioning, + serializer, + writeMetrics) + metrics("numPartitions").set(dep.partitioner.numPartitions) + val executionId = sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY) + SQLMetrics.postDriverMetricUpdates( + sparkContext, executionId, metrics("numPartitions") :: Nil) + dep + } } protected override def doExecute(): RDD[InternalRow] = { diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala new file mode 100644 index 0000000000000..114d3974bb0ee --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala @@ -0,0 +1,435 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.execution.metric + +import scala.collection.mutable +import scala.reflect.ClassTag +import scala.util.control.NonFatal + +import org.apache.spark.SparkContext +import org.apache.spark.internal.{LogEntry, Logging} +import org.apache.spark.sql.Dataset +import org.apache.spark.sql.execution.{BaseSubqueryExec, QueryExecution, SparkPlan, SubqueryAdaptiveBroadcastExec, SubqueryBroadcastExec, SubqueryExec, WholeStageCodegenExec} +import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper +import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, BroadcastExchangeLike, ReusedExchangeExec, ShuffleExchangeExec, ShuffleExchangeLike} +import org.apache.spark.util.{AccumulatorV2, LastAttemptAccumulator} + +/* + * SQLLastAttemptAccumulator is a LastAttemptAccumulator that allows tracking the last attempt + * updates that happened in the scope of execution of a plan created by a specific Dataset's + * QueryExecution. + * + * Tracking RDDs belonging to a Dataset execution. + * ----------------------------------------------- + * Dataset executes executedPlan from its QueryExecution. Each SparkPlan node in the + * executedPlan saves the RDD with its execution (executeRDD or executeColumnarRDD). However, + * the root RDD of a Spark Stage that actually gets submitted and executed is not necessarily + * that RDD. It may be an ephemeral RDD created on the fly when submitting the job, e.g.: + * - for result stages, there may be additional transformations to format the results, like + * apply an Encoder (e.g. turn InternalRows into Rows) + * or transformations for Arrow + * - for map stages, there may be some additional transformations to prepare the shuffle + * data in correct format. + * - operations like dataframe caching may wrap the plan results to format and write to cache. + * We therefore cannot track the metrics updates just by RDD id. However, each SparkPlan also + * creates an RDDOperationScope, and wraps the execution it submits by that scope. + * The completed Tasks should have RDDOperationScope of the SparkPlan that submitted the + * Stage. We need to extract the RDDOperationScopes from Dataset.queryExecution.executedPlan + * to track last attempt metric updates coming from that execution. + * + * Additionally, it is possible that the same queryExecution.executedPlan is reused. For + * example, when collect() is called multiple times on the same Dataset. + * - Part of the execution (e.g. the shuffles) should then be reused. Accumulator should still + * keep their partial values associated to its RDDOperationScope, and return it for this + * new attempt. + * - Some of the execution (e.g. the result stage) may be recomputed. Since the SparkPlan will + * be the same, RDDOperationScope will be the same, and this should become a newer execution + * of the same RDD, which should replace the previous one. + * + * AQE plan changes + * ---------------- + * AQE re-optimizes LogicalPlan and creates new SparkPlan. If the new plan doesn't contain + * some of the QueryStages from the previous plan, they can be cancelled while they already + * started running and accumulated some metric results. + * If the metric is part of SparkPlan.metrics, then the newly created plan will have new + * metrics and the old metrics would have been discarded; so nothing needs to be tracked here. + * But if the metric is coming from outside, it can be reused by the new SparkPlan. + * A new plan will have a new RDD and a new RDDOperationScope, so by tracking these for the + * final AQE plan, only values from the final plan and execution should be aggregated. + * + * It can also happen that the new AQE plan reuses SparkPlan instances from the old plan, + * see CancelShuffleStageInBroadcastJoin. However, in that case, the old plan will be put + * under some new plan in newly submitted Stages. Since we only truly track the plans that + * submit Stages, these should be different and enough to disambiguate. + * + * Driver only updates + * ------------------- + * The metric can be updated directly on the driver side, during the execution of catalyst + * optimizer. One example is [[ConvertToLocalRelation]] optimization rule, which constant folds + * pieces of the plan. + * Execution in this scope is tagged with [[QueryExecution.id]] using + * [[SparkContext.DATASET_QUERY_EXECUTION_ID_KEY]] property, and this metric is tracking + * the metric value separately for each QueryExecution. + * Like with LastAttemptAccumulator, the metric will bail out if it's updated both from the driver + * and from executor Tasks. + * + * Cached / Checkpointed plans + * --------------------------- + * If the metric was used inside a cached (df.cache, df.persist) or checkpointed (df.checkpoint, + * df.localCheckpoint) plan, which is then turned into an RDDScanExec or InMemoryTableScanExec + * in the Dataset's executedPlan, [[lastAttemptValueForDataset]] and + * [[lastAttemptValueForQueryExecution]] are declared undefined behavior. In this case, + * [[lastAttemptValueForHighestRDDId()]] should be used instead, which returns the value from + * the execution in which the plan was cached/checkpointed. + * + * The main issue is if the metric is in the top stage of the cached plan. When that plan is + * executed in some Dataset (as lazy execution), the metric will be executed in the scope of the + * stage that contains the InMemoryTableScanExec / RDDScanExec, which will be some parent of that + * plan, and not plan of the cached plan. So if the cached plan is then used in another Dataset, + * that Dataset will not have information about that parent. + * There could be some hacks done to fix it by recording in the InMemoryRelation the scopes in + * which it was materialized. There are also other issues, like that checkpoint throws away the + * plan, so it would also have to record the RDD scopes used during checkpointing. This gets + * further complicated if recomputations are involved, and are done in yet another scope. + * It was declared undefined behavior instead of pursuing this. + */ + +/** + * A trait that can be mixed into a subclass of [[AccumulatorV2]] to track the "logical" + * value of the "last attempt" of the execution using the accumulator. + * In addition to what [[LastAttemptAccumulator]] does, it allows tracking the last attempt + * executed in the scope of a Dataset's QueryExecution, via + * [[lastAttemptValueForDataset]] and [[lastAttemptValueForQueryExecution]] methods. + */ +trait SQLLastAttemptAccumulator[IN, OUT, PARTIAL, DRIVER_ACC] + extends LastAttemptAccumulator[IN, OUT, PARTIAL] { + this: AccumulatorV2[IN, OUT] => + + /** Create a fresh accumulator to hold driver-side values for one QueryExecution. */ + protected def newDriverQueryExecutionAcc(): DRIVER_ACC + /** Add a value to a driver-side per-QueryExecution accumulator. */ + protected def addToDriverAcc(acc: DRIVER_ACC, value: IN): Unit + /** Set the value of a driver-side per-QueryExecution accumulator. */ + protected def setDriverAcc(acc: DRIVER_ACC, value: OUT): Unit + /** Read the value of a driver-side per-QueryExecution accumulator. */ + protected def driverAccValue(acc: DRIVER_ACC): OUT + + @transient + private var lastAttemptDirectDriverQueryExecutionValues: mutable.Map[String, DRIVER_ACC] = _ + + override def initializeLastAttemptAccumulator()(implicit ct: ClassTag[PARTIAL]): Unit = try { + super.initializeLastAttemptAccumulator()(ct) + lastAttemptDirectDriverQueryExecutionValues = new mutable.HashMap[String, DRIVER_ACC]() + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in initializeLastAttemptAccumulator", + exception = Some(e)) + } + + override def resetLastAttemptAccumulator(): Unit = try { + super.resetLastAttemptAccumulator() + lastAttemptDirectDriverQueryExecutionValues = new mutable.HashMap[String, DRIVER_ACC]() + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in resetLastAttemptAccumulator", + exception = Some(e)) + } + + override protected def assertValid() = { + super.assertValid() + assert(lastAttemptDirectDriverQueryExecutionValues != null) + } + + protected def getOrCreateDirectDriverQueryExecutionValue(queryExecutionId: String): DRIVER_ACC = { + lastAttemptDirectDriverQueryExecutionValues.synchronized { + if (!lastAttemptDirectDriverQueryExecutionValues.contains(queryExecutionId)) { + lastAttemptDirectDriverQueryExecutionValues.put( + queryExecutionId, newDriverQueryExecutionAcc()) + } + lastAttemptDirectDriverQueryExecutionValues(queryExecutionId) + } + } + + protected def getActiveDatasetQueryExecutionId: Option[String] = { + SparkContext + .getActive + .flatMap(sc => Option(sc.getLocalProperty(SparkContext.DATASET_QUERY_EXECUTION_ID_KEY))) + } + + /** + * Check if the value is added on the driver side, not from within a task. + * If it is set in the scope of a Dataset's QueryExecution, associate it with that scope. + * This must be called from `add` methods of any AccumulatorV2 subclass supporting + * SQL last attempt metrics to set what the `value` of the metric is after the operation. + * This should be called there after [[setValueIfOnDriverSide]]. + */ + protected def addQueryExecutionValueIfOnDriverSide(value: IN): Unit = try { + // Note: setValueIfOnDriverSide will already make it invalid if there are also RDD updates. + if (isAtDriverSide && lastAttemptAccumulatorInitialized && !lastAttemptAccumulatorInvalid) { + // Direct update on the driver, not from within a task. + getActiveDatasetQueryExecutionId match { + case Some(qeId) => + addToDriverAcc(getOrCreateDirectDriverQueryExecutionValue(qeId), value) + case None => // pass + } + } + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in addQueryExecutionValueIfOnDriverSide", + exception = Some(e)) + } + + /** + * Like [[addQueryExecutionValueIfOnDriverSide]], but for set operations. + */ + protected def setQueryExecutionValueIfOnDriverSide(value: OUT): Unit = try { + if (isAtDriverSide && lastAttemptAccumulatorInitialized && !lastAttemptAccumulatorInvalid) { + getActiveDatasetQueryExecutionId match { + case Some(qeId) => + setDriverAcc(getOrCreateDirectDriverQueryExecutionValue(qeId), value) + case None => // pass + } + } + } catch { + case NonFatal(e) => + unexpectedLastAttemptMetricOperation( + invalidate = true, + reason = "Unexpected exception in setQueryExecutionValueIfOnDriverSide", + exception = Some(e)) + } + + override def logAccumulatorState: LogEntry = try { + val driverQEVals = Option(lastAttemptDirectDriverQueryExecutionValues) + .map(_.map { case (key, acc) => s"$key -> ${driverAccValue(acc)}" }.mkString("\n")) + .getOrElse("") + super.logAccumulatorState + + log""" + |Direct driver QE values: + |${MDC(logKeyAccumulatorState, driverQEVals)} + """.stripMargin + } catch { + case NonFatal(e) => + logWarning(log"Unexpected exception in logAccumulatorState", e) + log"" + } + + /** + * Returns the last attempt value of this accumulator, aggregated from the last execution of this + * QueryExecution. + * + * @note The output of this method is undefined if this metric was used inside a part of the plan + * which was either checkpointed (e.g. df.localCheckpoint(), df.checkpoint()) or cached + * (e.g. df.cache(), df.persist()). + * [[lastAttemptValueForHighestRDDId()]] should be used instead, which returns the + * value from the execution in which the plan was cached/checkpointed. + * + * @return None if the last attempt value cannot be established, Some(value) otherwise. + */ + def lastAttemptValueForQueryExecution(qe: QueryExecution): Option[OUT] = { + if (lastAttemptAccumulatorInvalid) return None + assertValid() + // If there was a driver set value defined in the scope of this QueryExecution, return that. + lastAttemptDirectDriverQueryExecutionValues.get(qe.id.toString) match { + case Some(acc) => return Some(driverAccValue(acc)) + case None => // pass + } + // Otherwise, gather the RDD scopes from the plan and find metric updates from these scopes. + val scopes = SQLLastAttemptAccumulator.extractStageRDDScopes(qe.executedPlan) + scopes match { + case Left(bailOutReason) => + unexpectedLastAttemptMetricOperation( + invalidate = false, + reason = s"Unable to extract RDD scopes from query execution plan: $bailOutReason") + None + case Right(scopes) => + lastAttemptValueForRDDScopes(scopes) + } + } + + /** + * Returns the last attempt value of this accumulator, aggregated from the last execution of this + * Dataset. + * + * @note The output of this method is undefined if this metric was used inside a part of the plan + * which was either checkpointed (e.g. df.localCheckpoint(), df.checkpoint()) or cached + * (e.g. df.cache(), df.persist()). + * [[lastAttemptValueForHighestRDDId()]] should be used instead, which returns the + * value from the execution in which the plan was cached/checkpointed. + * + * @return None if the last attempt value cannot be established, Some(value) otherwise. + */ + def lastAttemptValueForDataset(ds: Dataset[_]): Option[OUT] = { + lastAttemptValueForQueryExecution(ds.queryExecution) + } + + /** Visible for testing. */ + def getDirectDriverQueryExecutionValue(qeId: String): Option[OUT] = { + lastAttemptDirectDriverQueryExecutionValues.get(qeId).map(driverAccValue) + } +} + +object SQLLastAttemptAccumulator extends Logging { + + private[metric] def extractStageRDDScopes(sparkPlan: SparkPlan): Either[String, Seq[String]] = { + var bailOutReason: Option[String] = None + + // recurse, setting the bailOutReason on failure, or returning the list of scopes on success. + def recurse(sparkPlan: SparkPlan): Seq[String] = { + if (bailOutReason.isDefined) { + Nil + } else { + extractStageRDDScopes(sparkPlan) match { + case Left(reason) => + bailOutReason = Some(reason) + Nil + case Right(scopes) => scopes + } + } + } + + def scopeIds(sparkPlan: SparkPlan): Seq[String] = { + AdaptiveSparkPlanHelper.stripAQEPlan(sparkPlan) match { + case w: WholeStageCodegenExec => + // WholeStageCodegenExec can fallback and execute the child plan without codegen instead, + // we don't know when this happens, so we need to account for both cases. + // It will never be both at the same time as this is a compilation time decision, so + // returning both won't result in duplicates. + Seq(w.rddScopeId, w.child.rddScopeId) + case p => Seq(p.rddScopeId) + } + } + + // The root of the plan is submitted as a result stage. + val resultStageScopes = scopeIds(sparkPlan) + + val stagesScopes = AdaptiveSparkPlanHelper.flatMap(sparkPlan) { + case _ if bailOutReason.isDefined => Nil + + // broadcast exchange stage submitting nodes + case bl: BroadcastExchangeLike => bl match { + case b: BroadcastExchangeExec => + // The job is submitted in scope of child of the broadcast exchange. + // ``` + // val rs = child.executeCollectResult() + // ``` + // <- executeCollectResult() is called on the child, and child executes it in its scope. + scopeIds(b.child) + case p => + // Bail out if future unknown implementation is encountered. + bailOutReason = Some(s"Unsupported BroadcastExchangeLike: ${p.getClass.getName}") + Nil + } + + // shuffle exchange stage submitting nodes + case sl: ShuffleExchangeLike => sl match { + // All shuffle exchange implementations create the ShuffledRowRDD / ShuffledBlockRDD + // with its own scope, and it will be executed in that scope. + case s: ShuffleExchangeExec => scopeIds(s) + case p => + // Bail out if future unknown implementation is encountered. + bailOutReason = Some(s"Unsupported ShuffleExchangeLike: ${p.getClass.getName}") + Nil + } + + // reused exchange + case r: ReusedExchangeExec => + // Reused exchange is going to reuse stuff executed in the scope of its child, + // i.e. the exchange it reuses. + recurse(r.child) + + case sl: BaseSubqueryExec => sl match { + case s: SubqueryExec => + // ``` + // val rows: Array[InternalRow] = if (maxNumRows.isDefined) { + // child.executeTake(maxNumRows.get) + // } else { + // child.executeCollect() + // } + // ``` + // will launch stages in scope of child. + scopeIds(s.child) + case _: SubqueryBroadcastExec => + // Used by DPP filter only, not part of main flow of query execution. + Nil + case _: SubqueryAdaptiveBroadcastExec => + // Used by DPP filter only. + Nil + case p => + // Bail out if future unknown implementation is encountered. + bailOutReason = Some(s"Unsupported BaseSubqueryExec: ${p.getClass.getName}") + Nil + } + + /* Useful comments for posterity. + // cached table node + case _: InMemoryTableScanLike => + // Do nothing for cached tables. There are many border cases where it wouldn't work. + // Some notes for posterity: + // For [[InMemoryTableScanExec]], we could recursed into the cachedPlan, but: + // - if the metric is in the top stage of that plan, then it would be executed in the scope + // of the stage of whatever execution that InMemoryTableScanExec is part of when the + // plan is cached. [[InMemoryTableScanExec]] is not a stage submitting node by itself, and + // by itself it doesn't have visibility into the parent that submits the stage that + // materializes the cache. If the current executedPlan is not the one that materializes, + // then the metric would return 0 instead of the value from the cached execution. If the + // current executedPlan is the one that materializes the cache, then it would be the + // correct value. + // - if the metric is in a map stage of the cachedPlan, then it would be correctly + // annotated with the scope of that stage, and it would work correctly. + // + // Since it's hard to achieve a consistent behavior here, we just do not support it. + Nil + + // RDD node + case _: RDDScanExec => + // Similar as with cached tables, do nothing with RDDs. + // This could be a plan coming from an execution of df.checkpoint(). + // Since checkpointing cuts the references to the original plan, there is no way to descend + // into it to check attribution. + // We could try to make checkpoint collect and store the scopes of the original execution, + // but even then it would face similar inconsistencies as described above for cached plans. + // - if the metric is in the top stage of that plan, then if it was executed in the scope + // of this execution, it would be attributed to the scope of the parent stage that is + // consuming the checkpointed RDD, not to any scope of the original plan. + // - if the metric is in a map stage of the plan that was checkpointed, it requires that + // checkpoint would track these stages and scopes. + // + // Since it's hard to achieve a consistent behavior here, we just do not support it. + Nil + */ + + case _ => Nil // only extract from nodes that submit stages + } + + // also collect the plan scopes of all subqueries, which are executed "on the side". + val subqueriesScopes = AdaptiveSparkPlanHelper.flatMap(sparkPlan) { p => + p.subqueries.flatMap(recurse) + } + + if (bailOutReason.isDefined) { + Left(bailOutReason.get) + } else { + Right(resultStageScopes ++ stagesScopes ++ subqueriesScopes) + } + } +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetric.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetric.scala new file mode 100644 index 0000000000000..33326fb8e5bc4 --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetric.scala @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.execution.metric + +import org.apache.spark.SparkContext +import org.apache.spark.util.AccumulatorV2 + +class SQLLastAttemptMetric( + metricType: String, + initValue: Long = 0L) + extends SQLMetric(metricType, initValue) + with SQLLastAttemptAccumulator[Long, Long, Long, SQLMetric] { + + override protected def partialMergeVal: Long = _value + + override protected def partialMerge(value: Long): Unit = { + // For SQLLastAttemptMetric, this is just add to the underlying SQLMetric. + super.add(value) + } + + override protected def isMergeable(other: AccumulatorV2[_, _]): Boolean = other match { + case o: SQLLastAttemptMetric => o.metricType == metricType + case _ => false + } + + // SQLLastAttemptMetric is used internally to aggregate system metrics (counters) such as + // number of rows processed, and it should not store user data. + protected def accumulatorStoresUserData: Boolean = false + + override protected def newDriverQueryExecutionAcc(): SQLMetric = + new SQLMetric(metricType, initValue) + override protected def addToDriverAcc(acc: SQLMetric, value: Long): Unit = acc.add(value) + override protected def setDriverAcc(acc: SQLMetric, value: Long): Unit = acc.set(value) + override protected def driverAccValue(acc: SQLMetric): Long = acc.value + + override def copy(): SQLLastAttemptMetric = { + val newAcc = new SQLLastAttemptMetric(metricType, initValue) + newAcc._value = _value + newAcc + } + + override def add(v: Long): Unit = { + super.add(v) + if (v >= 0) { + // set value of SQLMetric after the add. + setValueIfOnDriverSide(value) + addQueryExecutionValueIfOnDriverSide(v) + } + } + + override def set(v: Long): Unit = { + super.set(v) + if (v >= 0) { + // set value of SQLMetric after the set. + setValueIfOnDriverSide(value) + setQueryExecutionValueIfOnDriverSide(value) + } + } + +} + +object SQLLastAttemptMetrics { + /** + * Create a metric to report the value aggregated from the last attempt of each task. These + * would be the values for the tasks that actually contributed to the final output of the + * execution. + */ + def createMetric(sc: SparkContext, name: String): SQLLastAttemptMetric = { + val acc = new SQLLastAttemptMetric(SQLMetrics.SUM_METRIC) + acc.register(sc, name = SQLMetrics.metricsCache.get(name), countFailedValues = false) + acc.initializeLastAttemptAccumulator() + acc + } +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala index 13f4d7926bea8..0523df282cda5 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala @@ -45,7 +45,7 @@ class SQLMetric( // for SPARK-11013. assert(initValue <= 0) // _value will always be either initValue or non-negative. - private var _value = initValue + private[metric] var _value = initValue override def copy(): SQLMetric = { val newAcc = new SQLMetric(metricType, initValue) @@ -110,7 +110,7 @@ class SQLMetric( } object SQLMetrics { - private val SUM_METRIC = "sum" + private[metric] val SUM_METRIC = "sum" private val SIZE_METRIC = "size" private val TIMING_METRIC = "timing" private val NS_TIMING_METRIC = "nsTiming" @@ -120,7 +120,7 @@ object SQLMetrics { val cachedSQLAccumIdentifier = Some(AccumulatorContext.SQL_ACCUM_IDENTIFIER) - private val metricsCache: LoadingCache[String, Option[String]] = + private[metric] val metricsCache: LoadingCache[String, Option[String]] = CacheBuilder.newBuilder().maximumSize(10000) .build(new CacheLoader[String, Option[String]] { override def load(name: String): Option[String] = { diff --git a/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala b/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala index 036ec943127d8..291aa7cab7256 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala @@ -300,6 +300,28 @@ trait QueryTestBase super.withSQLConf(pairs: _*)(f) } + /** + * Temporarily sets SparkContext configuration values for testing. + * This is for configs that must be set on the SparkContext (not + * SQLConf), such as testing flags. + */ + protected def withSparkContextConf[T]( + pairs: (String, String)*)(f: => T): T = { + val sc = spark.sparkContext + val oldValues = pairs.map { case (k, _) => + k -> sc.conf.getOption(k) + } + try { + pairs.foreach { case (k, v) => sc.conf.set(k, v) } + f + } finally { + oldValues.foreach { + case (k, Some(v)) => sc.conf.set(k, v) + case (k, None) => sc.conf.remove(k) + } + } + } + /** * Drops functions after calling `f`. A function is represented by (functionName, isTemporary). */ diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/MetricsFailureInjectionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/MetricsFailureInjectionSuite.scala new file mode 100644 index 0000000000000..847a12f4f305c --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/MetricsFailureInjectionSuite.scala @@ -0,0 +1,364 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.execution.metric + +import scala.util.Random + +import org.apache.spark.internal.config +import org.apache.spark.sql.{Column, Dataset} +import org.apache.spark.sql.execution.adaptive.{AQETestHelper, DisableAdaptiveExecutionSuite} +import org.apache.spark.sql.functions.udf +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.test.SharedSparkSession + +class MetricsFailureInjectionSuite + extends SharedSparkSession + with SQLMetricsTestUtils + // Need to control AQE per-test to ensure expected plan shapes. + with DisableAdaptiveExecutionSuite { + + import testImplicits._ + + override def beforeAll(): Unit = { + super.beforeAll() + // Disable re-use, since it interferes with the forced replanning. + spark.conf.set(SQLConf.EXCHANGE_REUSE_ENABLED, false) + } + + def setUpTestTable(tableName: String): Unit = { + val rand = new Random(1) + val randomPrefix = rand.nextString(30) + spark.range(300).map { id => + (id, (id % 5).toInt, randomPrefix + (id % 111)) + }.toDF("id", "low_cardinality_col", "large_col") + .write.format("parquet").saveAsTable(tableName) + val numRecords = spark.read.table(tableName).count() + assert(numRecords === 300) + } + + for { + useAQE <- BOOLEAN_DOMAIN + } test(s"Two stage metrics AQE cancellation injection - useAQE=$useAQE") { + withSQLConf( + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> useAQE.toString) { + val stage1Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 1 counter") + val stage2Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 2 counter") + val stage1SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 1 SLAM") + val stage2SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 2 SLAM") + + def runQueryWithMetrics( + triggerMetrics: SQLMetric*)( + postRunChecks: Dataset[_] => Unit): Unit = { + assert(stage1Metric.value === 0) + assert(stage2Metric.value === 0) + withTable("test_table") { + setUpTestTable("test_table") + AQETestHelper.withForcedCancellation(triggerMetrics: _*) { + val stage1MetricsExpr = incrementMetrics(Seq(stage1Metric, stage1SLAMetric)) + val stage1 = spark.read.table("test_table").filter(Column(stage1MetricsExpr)) + val stage2MetricsExpr = incrementMetrics(Seq(stage2Metric, stage2SLAMetric)) + val stage2 = + stage1.groupBy("low_cardinality_col").count().filter(Column(stage2MetricsExpr)) + val finalDf = stage2.as[(Int, Long)] + val result = finalDf.collect() + + assert(result.toMap === (0 until 5).map(v => (v, 300 / 5)).toMap) + postRunChecks(finalDf) + stage1Metric.reset() + stage2Metric.reset() + } + } + } + + // SLAM values don't change with retries, so we can reuse the same assertions for all cases. + def assertSLAM(finalDf: Dataset[_]): Unit = { + assert(stage1SLAMetric.lastAttemptValueForHighestRDDId() === Some(300)) + assert(stage2SLAMetric.lastAttemptValueForHighestRDDId() === Some(5)) + + assert(stage1SLAMetric.lastAttemptValueForDataset(finalDf) === Some(300)) + assert(stage2SLAMetric.lastAttemptValueForDataset(finalDf) === Some(5)) + } + + // Case 1: No forced replanning. + runQueryWithMetrics() { finalDf => + assert(stage1Metric.value === 300) + assert(stage2Metric.value === 5) + + assertSLAM(finalDf) + } + + // Case 2: Replan on stage1Metric. + runQueryWithMetrics(stage1Metric) { finalDf => + if (useAQE) { + assert(stage1Metric.value > 300) + } else { + assert(stage1Metric.value === 300) + } + assert(stage2Metric.value === 5) + + assertSLAM(finalDf) + } + + // Case 3: Replan on stage2Metric (will be ignored, because this is a result stage). + runQueryWithMetrics(stage2Metric) { finalDf => + assert(stage1Metric.value === 300) + assert(stage2Metric.value === 5) + + assertSLAM(finalDf) + } + + // Case 4: Replan on both metrics (only first will actually trigger). + runQueryWithMetrics(stage1Metric, stage2Metric) { finalDf => + if (useAQE) { + assert(stage1Metric.value > 300) + } else { + assert(stage1Metric.value === 300) + } + assert(stage2Metric.value === 5) + + assertSLAM(finalDf) + } + } + } + + for { + useAQE <- BOOLEAN_DOMAIN + } test(s"Three stage metrics AQE cancellation injection - useAQE=$useAQE") { + withSQLConf( + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> useAQE.toString) { + val stage1Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 1 counter") + val stage2Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 2 counter") + val stage3Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 3 counter") + val stage1SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 1 SLAM") + val stage2SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 2 SLAM") + val stage3SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 3 SLAM") + + def runQueryWithMetrics( + triggerMetrics: SQLMetric*)(postRunChecks: Dataset[_] => Unit): Unit = { + assert(stage1Metric.value === 0) + assert(stage2Metric.value === 0) + withTable("primary_table", "secondary_table") { + // Use the same layout for both. Makes the query a non-obvious self-join essentially. + setUpTestTable("primary_table") + setUpTestTable("secondary_table") + AQETestHelper.withForcedCancellation(triggerMetrics: _*) { + val stage1MetricsExpr = incrementMetrics(Seq(stage1Metric, stage1SLAMetric)) + val stage1 = spark.read.table("primary_table") + .filter(Column(stage1MetricsExpr)) + val stage2MetricsExpr = incrementMetrics(Seq(stage2Metric, stage2SLAMetric)) + val stage2 = stage1.join( + spark.read.table("secondary_table"), + usingColumn = "id", + joinType = "fullOuter") + .filter(Column(stage2MetricsExpr)) + val stage3MetricsExpr = incrementMetrics(Seq(stage3Metric, stage3SLAMetric)) + val stage3 = stage2 + .groupBy("primary_table.low_cardinality_col") + .count() + .filter(Column(stage3MetricsExpr)) + val finalDf = stage3.as[(Int, Long)] + val result = finalDf.collect() + assert(result.toMap === (0 until 5).map(v => (v, 300 / 5)).toMap) + postRunChecks(finalDf) + stage1Metric.reset() + stage2Metric.reset() + stage3Metric.reset() + } + } + } + + // SLAM values don't change with retries, so we can reuse the same assertions for all cases. + def assertSLAM(finalDf: Dataset[_]): Unit = { + assert(stage1SLAMetric.lastAttemptValueForHighestRDDId() === Some(300)) + assert(stage2SLAMetric.lastAttemptValueForHighestRDDId() === Some(300)) + assert(stage3SLAMetric.lastAttemptValueForHighestRDDId() === Some(5)) + + assert(stage1SLAMetric.lastAttemptValueForDataset(finalDf) === Some(300)) + assert(stage2SLAMetric.lastAttemptValueForDataset(finalDf) === Some(300)) + assert(stage3SLAMetric.lastAttemptValueForDataset(finalDf) === Some(5)) + } + + // Case 1: No forced replanning. + runQueryWithMetrics() { finalDf => + assert(stage1Metric.value === 300) + assert(stage2Metric.value === 300) + assert(stage3Metric.value === 5) + + assertSLAM(finalDf) + } + + // Case 2: Replan on stage1Metric. + runQueryWithMetrics(stage1Metric) { finalDf => + if (useAQE) { + assert(stage1Metric.value > 300) + } else { + assert(stage1Metric.value === 300) + } + assert(stage2Metric.value === 300) + assert(stage3Metric.value === 5) + + assertSLAM(finalDf) + } + + // Case 3: Replan on stage2Metric (will also re-run the first stage). + runQueryWithMetrics(stage2Metric) { finalDf => + if (useAQE) { + assert(stage1Metric.value > 300) + assert(stage2Metric.value > 300) + } else { + assert(stage1Metric.value === 300) + assert(stage2Metric.value === 300) + } + assert(stage3Metric.value === 5) + + assertSLAM(finalDf) + } + + // Case 4: Replan on all metrics (only first will actually trigger). + runQueryWithMetrics(stage1Metric, stage2Metric, stage3Metric) { finalDf => + if (useAQE) { + assert(stage1Metric.value > 300) + } else { + assert(stage1Metric.value === 300) + } + assert(stage2Metric.value === 300) + assert(stage3Metric.value === 5) + + assertSLAM(finalDf) + } + } + } + + for { + injectFailure <- BOOLEAN_DOMAIN + } test(s"Two stage metrics block failure injection - injectFailure=$injectFailure") { + val stage1Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 1 counter") + val stage2Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 2 counter") + val stage1SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 1 SLAM") + val stage2SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 2 SLAM") + + def runQueryWithMetrics( + triggerMetrics: SQLMetric*)(postRunChecks: Dataset[_] => Unit): Unit = { + assert(stage1Metric.value === 0) + assert(stage2Metric.value === 0) + withTable("test_table") { + setUpTestTable("test_table") + withSparkContextConf( + config.Tests.INJECT_SHUFFLE_FETCH_FAILURES.key -> injectFailure.toString) { + val stage1MetricsExpr = incrementMetrics(Seq(stage1Metric, stage1SLAMetric)) + val stage1 = spark.read.table("test_table").filter(Column(stage1MetricsExpr)) + val stage2MetricsExpr = incrementMetrics(Seq(stage2Metric, stage2SLAMetric)) + val stage2 = + stage1.groupBy("low_cardinality_col").count().filter(Column(stage2MetricsExpr)) + val finalDf = stage2.as[(Int, Long)] + val result = finalDf.collect() + assert(result.toMap === (0 until 5).map(v => (v, 300 / 5)).toMap) + postRunChecks(finalDf) + stage1Metric.reset() + stage2Metric.reset() + } + } + } + + runQueryWithMetrics() { finalDf => + if (injectFailure) { + assert(stage1Metric.value > 300) + } else { + assert(stage1Metric.value === 300) + } + // Stage2 doesn't have a downstream shuffle stage we can fail. + assert(stage2Metric.value === 5) + + assert(stage1SLAMetric.lastAttemptValueForHighestRDDId() === Some(300)) + assert(stage2SLAMetric.lastAttemptValueForHighestRDDId() === Some(5)) + + assert(stage1SLAMetric.lastAttemptValueForDataset(finalDf) === Some(300)) + assert(stage2SLAMetric.lastAttemptValueForDataset(finalDf) === Some(5)) + } + } + + for { + injectFailure <- BOOLEAN_DOMAIN + } test(s"Non-deterministic stage block failure injection - injectFailure=$injectFailure") { + val stage1Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 1 counter") + val stage2Metric = SQLMetrics.createMetric(spark.sparkContext, "stage 2 counter") + val stage1SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 1 SLAM") + val stage2SLAMetric = + SQLLastAttemptMetrics.createMetric(spark.sparkContext, "stage 2 SLAM") + + def runQueryWithMetrics( + triggerMetrics: SQLMetric*)(postRunChecks: Dataset[_] => Unit): Unit = { + assert(stage1Metric.value === 0) + assert(stage2Metric.value === 0) + withTable("test_table") { + setUpTestTable("test_table") + withSparkContextConf( + config.Tests.INJECT_SHUFFLE_FETCH_FAILURES.key -> injectFailure.toString) { + val stage1MetricsExpr = incrementMetrics(Seq(stage1Metric, stage1SLAMetric)) + val udfRand = + udf { + () => { + new Random().nextDouble() + } + }.asNondeterministic().apply().expr + val stage1 = spark.read.table("test_table") + .withColumn("non_deterministic_col", Column(udfRand)) + .filter(Column(stage1MetricsExpr)) + val stage2MetricsExpr = incrementMetrics(Seq(stage2Metric, stage2SLAMetric)) + val stage2 = stage1 + .groupBy("low_cardinality_col") + .avg("non_deterministic_col") + .filter(Column(stage2MetricsExpr)) + // Add an extra stage with a single task to avoid flaky failures. If a ResultTask + // returns non-deterministic results to the client, it forces the query to abort + // instead of retrying the input stages. + val finalDf = stage2.repartition(1).as[(Int, Double)] + val result = finalDf.collect() + // Don't compare the second value, since it's random. + assert(result.map(_._1).toSet === (0 until 5).toSet) + postRunChecks(finalDf) + stage1Metric.reset() + stage2Metric.reset() + } + } + } + + runQueryWithMetrics() { finalDf => + if (injectFailure) { + assert(stage1Metric.value > 300) + } else { + assert(stage1Metric.value === 300) + } + // Stage2 doesn't have a downstream shuffle stage we can fail. + assert(stage2Metric.value === 5) + + assert(stage1SLAMetric.lastAttemptValueForHighestRDDId() === Some(300)) + assert(stage2SLAMetric.lastAttemptValueForHighestRDDId() === Some(5)) + + assert(stage1SLAMetric.lastAttemptValueForDataset(finalDf) === Some(300)) + assert(stage2SLAMetric.lastAttemptValueForDataset(finalDf) === Some(5)) + } + } +} diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricIntegrationSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricIntegrationSuite.scala new file mode 100644 index 0000000000000..2e7af075a3e74 --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricIntegrationSuite.scala @@ -0,0 +1,705 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.execution.metric + +import org.apache.spark.internal.config +import org.apache.spark.sql.Column +import org.apache.spark.sql.catalyst.expressions.Literal +import org.apache.spark.sql.catalyst.optimizer.BuildRight +import org.apache.spark.sql.catalyst.plans.RightOuter +import org.apache.spark.sql.execution.{CoalescedPartitionSpec, CoalescedShuffleRead, WholeStageCodegenExec} +import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper +import org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoinExec +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.test.SharedSparkSession + +/** Tests [[SQLLastAttemptMetric]] used by [[RDD]]s and [[Dataset]]s */ +class SQLLastAttemptMetricIntegrationSuite + extends SharedSparkSession + with SQLMetricsTestUtils { + import testImplicits._ + + protected def withRetries = false + + test("single stage rdd updates with shared slam") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + val rdd1 = spark.sparkContext.parallelize(1 to 10, 2).map { x => + slam.add(1) + x + } + + rdd1.count() + assert(withRetries || slam.value === 10) + assert(slam.lastAttemptValueForAllRDDs() === Some(10)) + assert(slam.lastAttemptValueForRDDId(rdd1.id) === Some(10)) + assert(slam.lastAttemptValueForRDDIds(Seq(rdd1.id, rdd1.id)) === Some(10)) + assert(slam.lastAttemptValueForRDDIds(Seq(rdd1.id + 1, rdd1.id + 2)) === Some(0)) + assert(slam.lastAttemptValueForRDDIds(Seq(rdd1.id, rdd1.id + 10, rdd1.id)) === Some(10)) + assert(slam.lastAttemptValueForHighestRDDId() === Some(10)) + + val rdd2 = spark.sparkContext.parallelize(1 to 50, 3).map { x => + slam.add(3) + x + } + rdd2.count() + assert(withRetries || slam.value === 160) // +150 + assert(slam.lastAttemptValueForRDDId(rdd1.id) === Some(10)) // value for first rdd unaffected + assert(slam.lastAttemptValueForRDDId(rdd2.id) === Some(150)) // value for second rdd recorded + assert(slam.lastAttemptValueForAllRDDs() === Some(160)) // value for all rdds summed + assert(slam.getNumRDDs === 2) + assert(slam.lastAttemptValueForHighestRDDId() === Some(150)) // highest RDD id updated. + + // Re-executing rdd1 + rdd1.count() + assert(withRetries || slam.value === 170) // +10 + // Re-execution doesn't produce duplicate last attempt values + assert(slam.lastAttemptValueForRDDId(rdd1.id) === Some(10)) + assert(slam.lastAttemptValueForAllRDDs() === Some(160)) + assert(slam.getNumRDDs === 2) + // Highest RDD id tracks highest rdd.id, not the last RDD to be executed + assert(slam.lastAttemptValueForHighestRDDId() === Some(150)) + + // New RDD on top of rdd1, but in a single stage. + val rdd3 = rdd1.map { x => + slam.add(2) + x + } + rdd3.count() + assert(withRetries || slam.value === 200) // +30 + assert(slam.lastAttemptValueForRDDId(rdd1.id) === Some(10)) // stays the same + // The increment from rdd1 and rdd3 are in the same stage, so they are recorded together. + assert(slam.getNumRDDs === 3) + assert(slam.lastAttemptValueForRDDId(rdd3.id) === Some(30)) + assert(slam.getHighestRDDId === Some(rdd3.id)) + assert(slam.lastAttemptValueForHighestRDDId() === Some(30)) + + // Setting a value directly from the driver makes slam bail out, because it can't reason + // about what is the "last attempt" on driver vs. coming from RDD executions. + slam.set(42) + assert(!slam.getValid) + // Information stays available for logging and debugging. + assert(slam.getNumRDDs === 3) + assert(slam.getHighestRDDId === Some(rdd3.id)) + + logInfo(slam.logAccumulatorState) + } + + test("multi stage rdd updates") { + val slam1 = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "SLAM1") + val slam2 = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "SLAM2") + + val rdd1 = spark.sparkContext.parallelize(1 to 10, 2).map { x => + slam1.add(1) + x + } + val repartition = rdd1.repartition(10) + val rdd2 = repartition.map { x => + slam2.add(1) + x + } + rdd2.collect() + assert(withRetries || slam1.value === 10) + assert(withRetries || slam2.value === 10) + assert(slam1.lastAttemptValueForAllRDDs() === Some(10)) + assert(slam2.lastAttemptValueForAllRDDs() === Some(10)) + assert(slam1.lastAttemptValueForHighestRDDId() === Some(10)) + assert(slam2.lastAttemptValueForHighestRDDId() === Some(10)) + // It is executed in a Stage submitted by the repartition. + assert(slam1.lastAttemptValueForRDDId(rdd1.id) === Some(0)) + assert(slam1.lastAttemptValueForRDDId(repartition.id) === Some(0)) // Surprise, nope. + // Repartition creates a number of MapPartitionsRDDs, CoalescedRDDs, ShuffledRDDs... + // The actual stage that submits the map stage is somewhere internal. + assert(slam1.getHighestRDDId.isDefined) + val mapStageRddId = slam1.getHighestRDDId.get + assert(slam1.lastAttemptValueForRDDId(mapStageRddId) === Some(10)) + + // Test passing multiple ids. + assert(slam1.lastAttemptValueForRDDIds(Seq(rdd1.id, repartition.id)) === Some(0)) + assert(slam1.lastAttemptValueForRDDIds( + Seq(rdd1.id, mapStageRddId, repartition.id)) === Some(10)) + assert(slam1.lastAttemptValueForRDDIds(Seq(rdd1.id, rdd2.id)) === Some(0)) + assert(slam1.lastAttemptValueForRDDIds(Seq(-10)) === Some(0)) + + rdd2.collect() + // Repartition stage is reused, but result stage is re-executed. + assert(withRetries || slam1.value === 10) // no change + assert(withRetries || slam2.value === 20) // +10 + // Last attempt value is not duplicated, since result stage is an action on the same RDD. + assert(slam1.lastAttemptValueForAllRDDs() === Some(10)) + assert(slam2.lastAttemptValueForAllRDDs() === Some(10)) + + rdd1.collect() + assert(withRetries || slam1.value === 20) // +10 + // The first time around it was executed in the repartition RDD stage. + // This time around it is executed from action of rdd1. + assert(slam1.getNumRDDs === 2) + assert(slam1.lastAttemptValueForAllRDDs() === Some(20)) + assert(slam1.lastAttemptValueForRDDId(rdd1.id) === Some(10)) // new + assert(slam1.lastAttemptValueForRDDId(mapStageRddId) === Some(10)) // old + // Highest RDD id stays the same. + assert(slam1.getHighestRDDId === Some(mapStageRddId)) + assert(slam1.lastAttemptValueForHighestRDDId() === Some(10)) + + rdd1.collect() + assert(withRetries || slam1.value === 30) // +10 + // Still the same + assert(slam1.lastAttemptValueForAllRDDs() === Some(20)) + assert(slam1.lastAttemptValueForHighestRDDId() === Some(10)) + + rdd2.collect() + // Repartition stage is reused (again), but result stage is re-executed (again). + assert(withRetries || slam1.value === 30) // no change + assert(withRetries || slam2.value === 30) // +10 + // Last attempt value is not duplicated, since result stage is an action on the same RDD. + assert(slam1.lastAttemptValueForAllRDDs() === Some(20)) + assert(slam2.lastAttemptValueForAllRDDs() === Some(10)) + + // Executed in different RDDs, but never duplicated. + assert(slam1.lastAttemptValueForRDDId(rdd1.id) === Some(10)) + assert(slam1.lastAttemptValueForRDDId(mapStageRddId) === Some(10)) + assert(slam1.lastAttemptValueForRDDId(repartition.id) === Some(0)) + assert(slam1.lastAttemptValueForRDDId(rdd2.id) === Some(0)) + assert(slam2.lastAttemptValueForRDDId(rdd1.id) === Some(0)) + assert(slam2.lastAttemptValueForRDDId(mapStageRddId) === Some(0)) + assert(slam1.lastAttemptValueForRDDId(repartition.id) === Some(0)) + assert(slam2.lastAttemptValueForRDDId(rdd2.id) === Some(10)) + + val newRepartition = rdd1.repartition(10) + val newRdd2 = newRepartition.map { x => + slam2.add(1) + x + } + newRdd2.collect() + assert(withRetries || slam1.value === 40) // +10 + assert(withRetries || slam2.value === 40) // +10 + // SLAM metrics get re-executed in the new RDDs + // rdd1 is reused, but the shuffle is new, and that is what submits the map stage. + assert(slam1.getNumRDDs === 3) + assert(slam1.lastAttemptValueForAllRDDs() === Some(30)) // +10 + assert(slam2.getNumRDDs === 2) + assert(slam2.lastAttemptValueForAllRDDs() === Some(20)) // +10 + // Values are recorded for the new highest RDD id. + assert(slam1.getHighestRDDId.isDefined) + val newMapStageId = slam1.getHighestRDDId.get + assert(newMapStageId > mapStageRddId) + assert(slam2.getHighestRDDId === Some(newRdd2.id)) + assert(slam1.lastAttemptValueForRDDId(newMapStageId) === Some(10)) + assert(slam2.lastAttemptValueForRDDId(newRdd2.id) === Some(10)) + assert(slam1.lastAttemptValueForHighestRDDId() === Some(10)) + assert(slam2.lastAttemptValueForHighestRDDId() === Some(10)) + + logInfo(slam1.logAccumulatorState) + logInfo(slam2.logAccumulatorState) + } + + test("rdd take") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + val rdd = spark.sparkContext.parallelize(1 to 100, 100).map { x => + slam.add(1) + x + } + withSparkContextConf( + // make it fixed to not be affected by potential changes of default. + config.RDD_LIMIT_INITIAL_NUM_PARTITIONS.key -> "1", + config.RDD_LIMIT_SCALE_UP_FACTOR.key -> "4" + ) { + rdd.take(1) // execute 1 partition + assert(withRetries || slam.value === 1) + assert(slam.lastAttemptValueForAllRDDs() === Some(1)) + + // take(2) scales up from 1 partition; the exact number of partitions scanned + // depends on the scale-up algorithm. + val valueBefore = slam.value + rdd.take(2) + val slamAfterTake2 = slam.lastAttemptValueForAllRDDs() + assert(slamAfterTake2.isDefined) + assert(slamAfterTake2.get >= 2) // at least 2 partitions + assert(slamAfterTake2.get < 100) // but not all partitions. + + // take(100) should execute all 100 partitions + rdd.take(100) + assert(slam.lastAttemptValueForAllRDDs() === Some(100)) + assert(slam.getNumRDDs === 1) + } + + logInfo(slam.logAccumulatorState) + } + + test("rdd coalesce") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + val rdd1 = spark.sparkContext.parallelize(1 to 100, 100).map { x => + slam.add(1) + x + } + val rdd2 = rdd1.coalesce(20) + // Test that coalescing that changes partition count doesn't break anything. + rdd2.collect() + assert(slam.lastAttemptValueForRDDId(rdd2.id) === Some(100)) + rdd1.collect() + assert(slam.lastAttemptValueForRDDId(rdd1.id) === Some(100)) + + logInfo(slam.logAccumulatorState) + } + + test("dataset updates") { + val slam1 = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "SLAM1") + val df1 = spark.range(10).filter(Column(incrementMetric(slam1))) + + df1.collect() + assert(withRetries || slam1.value === 10) + assert(slam1.getHighestRDDId.isDefined) + val df1HighestId = slam1.getHighestRDDId.get + val df1ExecutedPlanRddId = df1.queryExecution.executedPlan.execute().id + assert(slam1.lastAttemptValueForHighestRDDId() === Some(10)) + assert(slam1.lastAttemptValueForRDDId(df1HighestId) === Some(10)) + // Values retrieved from the Dataset are the same as from the RDD. + assert(slam1.lastAttemptValueForDataset(df1) === Some(10)) + assert(slam1.lastAttemptValueForQueryExecution(df1.queryExecution) === Some(10)) + + df1.collect() + assert(withRetries || slam1.value === 20) // +10 + // The same executedPlan RDD is reused, but getByteArrayRdd creates a new wrapper. + assert(df1.queryExecution.executedPlan.execute().id === df1ExecutedPlanRddId) + assert(slam1.lastAttemptValueForHighestRDDId() === Some(10)) + // Both wrapper RDDs are summed in allRDDs. + assert(slam1.lastAttemptValueForAllRDDs() === Some(20)) + assert(slam1.lastAttemptValueForDataset(df1) === Some(10)) + + val df2 = df1.filter("id < 5").filter(Column(incrementMetric(slam1))) + df2.collect() + assert(withRetries || slam1.value === 35) // +15 + assert(slam1.getHighestRDDId.isDefined) + // Both incrementMetric expressions are within the same Stage, so they record together. + assert(slam1.lastAttemptValueForHighestRDDId() === Some(15)) + // allRDDs includes wrapper RDDs from repeated df1.collect() calls. + assert(slam1.lastAttemptValueForAllRDDs() === Some(35)) + // New Dataset records only new value. + assert(slam1.lastAttemptValueForDataset(df2) === Some(15)) + // Value df1 is still remembered. + assert(slam1.lastAttemptValueForDataset(df1) === Some(10)) + + val slam2 = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "SLAM2") + val df3 = df1.repartition(1).filter(Column(incrementMetric(slam2))) + df3.collect() + assert(withRetries || slam1.value === 45) // +10 + assert(withRetries || slam2.value === 10) // new + // df3 creates a new plan, and the plan / RDD from df1 is not reused. + assert(slam1.getHighestRDDId.isDefined) + val slam1HighestId = slam1.getHighestRDDId.get + assert(slam1HighestId != df1HighestId) // new plan, new RDD id + assert(slam1.lastAttemptValueForHighestRDDId() === Some(10)) // from new execution + assert(slam1.lastAttemptValueForRDDId(df1HighestId) === Some(10)) // from first exec of df1 + // allRDDs includes wrapper RDDs from repeated collects. + assert(slam1.lastAttemptValueForAllRDDs() === Some(45)) + assert(slam2.lastAttemptValueForAllRDDs() === Some(10)) + // slam1 and slam2 are both executed in df3 + assert(slam1.lastAttemptValueForDataset(df3) === Some(10)) + assert(slam2.lastAttemptValueForDataset(df3) === Some(10)) + // slam2 is not executed in df1 and df2. + assert(slam2.lastAttemptValueForDataset(df1) === Some(0)) + assert(slam2.lastAttemptValueForDataset(df2) === Some(0)) + // slam1 value from df1 and df2 are still remembered. + assert(slam1.lastAttemptValueForDataset(df1) === Some(10)) + assert(slam1.lastAttemptValueForDataset(df2) === Some(15)) + + // Plans and RDDs get reused (result stage is re-executed; shuffle stage is purely reused). + df3.collect() + // No change in dataset values. + assert(slam1.lastAttemptValueForDataset(df3) === Some(10)) + assert(slam2.lastAttemptValueForDataset(df3) === Some(10)) + assert(slam1.lastAttemptValueForDataset(df1) === Some(10)) + assert(slam1.lastAttemptValueForDataset(df2) === Some(15)) + + logInfo(slam1.logAccumulatorState) + logInfo(slam2.logAccumulatorState) + } + + test("dataset limit") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "SLAM1") + // 10 partitions of 10 elements each + val df = spark.range(0, 1000, 1, 10).filter(Column(incrementMetric(slam))) + var expectedMetricValue = 0 + var expectedSLAMValue = 0 + + // Note: this is sensitive to the internal implementation of LimitExec. + + df.take(5) + // One partition executed, local limit pushed into partition. + expectedMetricValue = 5 + expectedSLAMValue = 5 + assert(withRetries || slam.value === expectedMetricValue) + assert(slam.lastAttemptValueForHighestRDDId() === Some(expectedSLAMValue)) + // take(5) actually inline creates a new Dataset, with new executed plan + assert(slam.lastAttemptValueForDataset(df) === Some(0)) + + df.take(50) + // One partition executed, local limit pushed into partition. + expectedMetricValue += 50 + expectedSLAMValue = 50 + assert(withRetries || slam.value === expectedMetricValue) + // New SQL plan creates new RDDs, so this is seen as new execution. + assert(slam.lastAttemptValueForHighestRDDId() === Some(expectedSLAMValue)) + assert(slam.getNumRDDs === 2) + assert(slam.lastAttemptValueForAllRDDs() === Some(expectedMetricValue)) + // take(50) executes a different inline Dataset and plan. + assert(slam.lastAttemptValueForDataset(df) === Some(0)) + + df.take(220) + // Three partitions executed. + expectedMetricValue += 300 + expectedSLAMValue = 300 + assert(withRetries || slam.value === expectedMetricValue) + assert(slam.lastAttemptValueForHighestRDDId() === Some(expectedSLAMValue)) + assert(slam.getNumRDDs === 3) + assert(slam.lastAttemptValueForAllRDDs() === Some(expectedMetricValue)) + // take(220) executes a different inline Dataset and plan. + assert(slam.lastAttemptValueForDataset(df) === Some(0)) + + df.take(320) + // Five partitions executed. + expectedMetricValue += 500 + expectedSLAMValue = 500 + assert(withRetries || slam.value === expectedMetricValue) + assert(slam.lastAttemptValueForHighestRDDId() === Some(expectedSLAMValue)) + assert(slam.getNumRDDs === 4) + assert(slam.lastAttemptValueForAllRDDs() === Some(expectedMetricValue)) + // take(320) executes a different inline Dataset and plan. + assert(slam.lastAttemptValueForDataset(df) === Some(0)) + + df.take(1) + // One partition scanned, local limit pushed into partition. + expectedMetricValue += 1 + expectedSLAMValue = 1 + assert(withRetries || slam.value === expectedMetricValue) + // New RDD, so the value from new execution is back to 1. + assert(slam.lastAttemptValueForHighestRDDId() === Some(expectedSLAMValue)) + assert(slam.getNumRDDs === 5) + assert(slam.lastAttemptValueForAllRDDs() === Some(expectedMetricValue)) + // take(1) executes a different inline Dataset and plan. + assert(slam.lastAttemptValueForDataset(df) === Some(0)) + + logInfo(slam.logAccumulatorState) + } + + test("driver set value") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + slam.set(10) + + // Regular metric value + assert(withRetries || slam.value === 10) + + assert(slam.getDirectDriverValue === Some(10)) + // "Driver update" is returned under "highest" and "all" RDDs + assert(slam.lastAttemptValueForHighestRDDId() === Some(10)) + assert(slam.lastAttemptValueForAllRDDs() === Some(10)) + assert(slam.getNumRDDs === 0) + // When specific RDDs are requested, driver value is not returned. + assert(slam.lastAttemptValueForRDDId(42) === Some(0)) + assert(slam.lastAttemptValueForRDDIds(Seq(7, 42)) === Some(0)) + + // Incrementing works + slam.add(5) + assert(withRetries || slam.value === 15) + assert(slam.lastAttemptValueForHighestRDDId() === Some(15)) + assert(slam.lastAttemptValueForAllRDDs() === Some(15)) + assert(slam.getDirectDriverValue === Some(15)) + + // Negative increments are ignored by SQLMetric + slam.add(-3) + assert(withRetries || slam.value === 15) + assert(slam.lastAttemptValueForHighestRDDId() === Some(15)) + assert(slam.lastAttemptValueForAllRDDs() === Some(15)) + assert(slam.getDirectDriverValue === Some(15)) + + // Reset does not reset SLAM. + slam.reset() + assert(withRetries || slam.value === 0) + assert(slam.lastAttemptValueForHighestRDDId() === Some(15)) + assert(slam.lastAttemptValueForAllRDDs() === Some(15)) + assert(slam.getDirectDriverValue === Some(15)) + + // Setting it back... + slam.set(20) + assert(withRetries || slam.value === 20) + assert(slam.lastAttemptValueForHighestRDDId() === Some(20)) + assert(slam.lastAttemptValueForAllRDDs() === Some(20)) + assert(slam.getDirectDriverValue === Some(20)) + assert(slam.getNumRDDs === 0) + + val df = spark.range(10).filter(Column(incrementMetric(slam))) + // SLAM was not executed in this Dataset, the driver value set manually + // before should not be returned. + assert(slam.lastAttemptValueForDataset(df) === Some(0)) + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === None) + df.collect() + assert(withRetries || slam.value === 30) + // SLAM bails out when it sees both driver and executor values + assert(slam.lastAttemptValueForHighestRDDId() === None) + assert(slam.lastAttemptValueForAllRDDs() === None) + assert(slam.lastAttemptValueForRDDId(42) === None) + assert(slam.lastAttemptValueForRDDIds(Seq(7, 42)) === None) + assert(slam.lastAttemptValueForDataset(df) === None) + assert(!slam.getValid) + assert(slam.getNumRDDs === 0) // invalidated before RDD got recorded + assert(slam.getDirectDriverValue === Some(20)) + // Invalidated before QueryExecution value was recorded. + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === None) + + slam.reset() + slam.set(10) + assert(withRetries || slam.value === 10) + // SLAM stays bailed out. + assert(slam.lastAttemptValueForHighestRDDId() === None) + assert(slam.lastAttemptValueForAllRDDs() === None) + assert(slam.lastAttemptValueForRDDId(42) === None) + assert(slam.lastAttemptValueForRDDIds(Seq(7, 42)) === None) + assert(slam.lastAttemptValueForDataset(df) === None) + assert(!slam.getValid) + // SLAM info doesn't get updated anymore when invalid, but stays around for debugging purposes. + assert(slam.getNumRDDs === 0) + assert(slam.getDirectDriverValue === Some(20)) + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === None) + + logInfo(slam.logAccumulatorState) + + // resetLastAttemptAccumulator resets it and makes it valid to be used again. + slam.resetLastAttemptAccumulator() + assert(slam.getValid) + slam.set(42) + assert(slam.lastAttemptValueForHighestRDDId() === Some(42)) + assert(slam.getDirectDriverValue === Some(42)) + assert(slam.getNumRDDs === 0) + assert(slam.lastAttemptValueForDataset(df) === Some(0)) + } + + test("ConvertToLocalRelation direct driver execution") { + // Normally ConvertToLocalRelation is disabled in tests. + withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> "") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + val df = Seq(1, 2, 3).toDF("a").filter(Column(incrementMetric(slam))) + + // SLAM is executed on the driver in the Optimized by ConvertToLocalRelation + df.collect() + assert(slam.lastAttemptValueForAllRDDs() === Some(3)) + assert(slam.lastAttemptValueForHighestRDDId() === Some(3)) + assert(slam.getDirectDriverValue === Some(3)) + // SLAM recognizes it was executed on the driver + // in the scope of the QueryExecution of this Dataset. + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === Some(3)) + + // Second action does not re-execute Optimizer. + df.collect() + assert(slam.lastAttemptValueForAllRDDs() === Some(3)) + assert(slam.lastAttemptValueForHighestRDDId() === Some(3)) + assert(slam.getDirectDriverValue === Some(3)) + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + + // Limitation: When a new Dataset is built and Optimizer reexecutes ConvertToLocalRelation, + // SLAM RDD retrieval cannot reason about re-execution on the driver, + // leading to duplicated metrics. + val df2 = df.withColumn("foo", Column(Literal("foo"))) + df2.collect() + assert(slam.lastAttemptValueForAllRDDs() === Some(6)) + assert(slam.lastAttemptValueForHighestRDDId() === Some(6)) + assert(slam.getDirectDriverValue === Some(6)) + // But it recognizes that it is done in a new QueryExecution and is able to distinguish that + // without duplicates. + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + assert(slam.lastAttemptValueForDataset(df2) === Some(3)) + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === Some(3)) + assert(slam.getDirectDriverQueryExecutionValue(df2.queryExecution.id.toString) === Some(3)) + + // No RDD executions were recorded. + assert(slam.getNumRDDs === 0) + + logInfo(slam.logAccumulatorState) + } + } + + test("ConvertToLocalRelation manual optimizer triggering") { + // Normally ConvertToLocalRelation is disabled in tests. + withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> "") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + val df = Seq(1, 2, 3).toDF("a").filter(Column(incrementMetric(slam))) + // Trigger the optimizer manually, which will trigger ConvertToLocalRelation + df.queryExecution.assertOptimized() + + // SLAM recognizes it was executed on the driver + // in the scope of the QueryExecution of this Dataset. + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === Some(3)) + + // Repeated actions do not re-execute Optimizer. + df.collect() + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + df.collect() + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + + logInfo(slam.logAccumulatorState) + } + } + + test("ConvertToLocalRelation in explain") { + // Normally ConvertToLocalRelation is disabled in tests. + withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> "") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + val df = Seq(1, 2, 3).toDF("a").filter(Column(incrementMetric(slam))) + + // EXPLAIN triggers the optimizer and triggered ConvertToLocalRelation to execute + df.explain(true) + assert(withRetries || slam.value === 3) + assert(slam.lastAttemptValueForAllRDDs() === Some(3)) + assert(slam.lastAttemptValueForHighestRDDId() === Some(3)) + assert(slam.getDirectDriverValue === Some(3)) + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === Some(3)) + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + // Retriggering EXPLAIN does not cause duplicates + df.explain(true) + assert(withRetries || slam.value === 3) + assert(slam.lastAttemptValueForAllRDDs() === Some(3)) + assert(slam.lastAttemptValueForHighestRDDId() === Some(3)) + assert(slam.getDirectDriverValue === Some(3)) + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === Some(3)) + + // Execution does not re-execute Optimizer and does not duplicate metric. + df.collect() + assert(withRetries || slam.value === 3) + assert(slam.lastAttemptValueForAllRDDs() === Some(3)) + assert(slam.lastAttemptValueForHighestRDDId() === Some(3)) + assert(slam.getDirectDriverValue === Some(3)) + assert(slam.lastAttemptValueForDataset(df) === Some(3)) + assert(slam.getDirectDriverQueryExecutionValue(df.queryExecution.id.toString) === Some(3)) + + // No RDD executions were recorded. + assert(slam.getNumRDDs === 0) + + logInfo(slam.logAccumulatorState) + } + } + + test("BroadcastNestedLoopJoin outer executes probe side twice") { + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + val build = + spark.range(5).selectExpr("id as b").hint("broadcast") + val probe = spark.range(100).selectExpr("id as p").filter(Column(incrementMetric(slam))) + val df = probe.join(build, usingColumns = Seq(), joinType = "rightouter") + df.collect() + assert(AdaptiveSparkPlanHelper.exists(df.queryExecution.executedPlan) { + case BroadcastNestedLoopJoinExec(_, _, BuildRight, RightOuter, None) => true + case _ => false + }) + // When build side is outer, probe side gets executed twice by BNLJ: + // once for matches, and once to mark unmatched build rows. + // This is a non-determinism correctness issue, and the two executions + // should not be double-counted in the last attempt value. + assert(slam.getNumRDDs === 2) + assert(slam.lastAttemptValueForAllRDDs() === Some(200)) + // The two executions are different RDDs, but only one of them is highest id. + assert(slam.lastAttemptValueForHighestRDDId() === Some(100)) + // Dataset dedups per scope and returns only the latest RDD's value. + assert(slam.lastAttemptValueForDataset(df) === Some(100)) + } + + test("SLAM with AQE CoalesceShufflePartitions") { + // Adapted from tests in CoalesceShufflePartitionsSuite + + val stage1Slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "SLAM2") + val stage2Slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "SLAM2") + val stage1MetricExpr = Column(incrementMetric(stage1Slam)) + val stage2MetricExpr = Column(incrementMetric(stage2Slam)) + + // Dataframe with a SLAM before and after a shuffle. + val df = spark.range(0, 1000, 1, numPartitions = 10) + .selectExpr("id % 20 as key", "id as value") + .filter(stage1MetricExpr) + .groupBy("key").count() + .filter(stage2MetricExpr) + + withSQLConf( + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true", + SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES.key -> "2000") { + df.collect() + } + + // Verify the AQE coalescing happened and coalesced the shuffle into 3 partitions. + // (based on ADVISORY_PARTITION_SIZE_IN_BYTES config) + val finalPlan = AdaptiveSparkPlanHelper.stripAQEPlan(df.queryExecution.executedPlan) + val shuffleReads = finalPlan.collect { + case r @ CoalescedShuffleRead() => r + } + assert(shuffleReads.nonEmpty) + shuffleReads.foreach { read => + // check there is actual coalescing of partitions happening + assert(read.isCoalescedRead) + assert(read.partitionSpecs.exists { + case p: CoalescedPartitionSpec if p.startReducerIndex < p.endReducerIndex - 1 => true + case _ => false + }) + } + + // Verify SLAM metrics. + assert(stage1Slam.lastAttemptValueForHighestRDDId() === Some(1000)) + assert(stage2Slam.lastAttemptValueForHighestRDDId() === Some(20)) + assert(stage1Slam.lastAttemptValueForDataset(df) === Some(1000)) + assert(stage2Slam.lastAttemptValueForDataset(df) === Some(20)) + } + + test("WholeStageCodegenExec fallback to non-codegen") { + withSQLConf( + SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "true", + SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key -> "1" // force fallback due to too large method + ) { + // This test is to verify that SLAM works correctly when WholeStageCodegenExec falls back + // to non-codegen execution. + val slam = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + val df = spark + .range(10) + .filter(Column(incrementMetric(slam))) + // these two operators will be turned into a WholeStageCodegen, + .selectExpr("id + 1 as foo", "id + 2 as bar") + .filter("foo < bar") + df.collect() + assert(slam.lastAttemptValueForDataset(df) === Some(10)) + // Metric is attributed to the child of the WSCG node. + val wscg = df.queryExecution.executedPlan.collectFirst { + case w: WholeStageCodegenExec => w + } + assert(wscg.isDefined) + assert(slam.getHighestRDDId.isDefined) + } + } +} + +class SQLLastAttemptMetricIntegrationSuiteWithStageRetries + extends SQLLastAttemptMetricIntegrationSuite { + override protected def withRetries = true + + override protected def test( + testName: String, + testTags: org.scalatest.Tag*) + (testFun: => Any) + (implicit pos: org.scalactic.source.Position): Unit = { + super.test(testName, testTags : _*) { + withSparkContextConf(config.Tests.INJECT_SHUFFLE_FETCH_FAILURES.key -> "true") { + // Stage retries should not affect SLAM metrics. + testFun + } + }(pos) + } +} diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricPlanShapesSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricPlanShapesSuite.scala new file mode 100644 index 0000000000000..ea8d9568f7e4b --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricPlanShapesSuite.scala @@ -0,0 +1,490 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.execution.metric + +import scala.reflect.ClassTag +import scala.util.Random + +import org.scalatest.Tag + +import org.apache.spark.internal.config +import org.apache.spark.sql.execution.{CollectLimitExec, RDDScanExec, SparkPlan} +import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, AQETestHelper, DisableAdaptiveExecutionSuite} +import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec +import org.apache.spark.sql.execution.exchange._ +import org.apache.spark.sql.functions.udf +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.test.SharedSparkSession + +class SQLLastAttemptMetricPlanShapesSuite + extends SharedSparkSession + with SQLMetricsTestUtils + // Need to control AQE per-test to ensure expected plan shapes. + with DisableAdaptiveExecutionSuite { + + import testImplicits._ + + import SQLLastAttemptMetricPlanShapesSuite._ + + // Avoid initialising this before the Spark Context is initialised. + protected var testSLAMetric: SQLLastAttemptMetric = _ + + protected def setUpTestTable(): Unit = { + val rand = new Random(1) + val randomPrefix = rand.nextString(30) + spark + .range(NUM_RECORDS) + .map { id => + (id, (id % LOW_CARDINALITY).toInt, randomPrefix + (id % LARGE_CARDINALITY)) + }.toDF("id", "low_cardinality_col", "large_col") + .write.format("parquet").saveAsTable(TABLE_NAME) + val numRecords = spark.read.table(TABLE_NAME).count() + assert(numRecords === 300) + } + + override protected def beforeAll(): Unit = { + super.beforeAll() + setUpTestTable() + testSLAMetric = SQLLastAttemptMetrics.createMetric(spark.sparkContext, "test SLAM") + // Move this into a local field so the closure doesn't hang on to the whole `this` + // reference as well. + val metric = testSLAMetric + val incrementMetric = () => { metric += 1; true } + val incrementMetricUdf = udf(incrementMetric).asNondeterministic() + spark.udf.register("increment_metric", incrementMetricUdf) + } + + override protected def afterAll(): Unit = { + spark.sql(s"DROP TABLE IF EXISTS $TABLE_NAME") + super.afterAll() + } + + override protected def beforeEach(): Unit = { + super.beforeEach() + // note: reset() does not influence lastAttemptValue, but influences regular value + testSLAMetric.reset() + } + + object MetricValue { + type Check = Option[Long] => Unit + + // Having the asserts in these helpers instead of in testPhysicalPlanShape + // produces better error messages. + def exactly(expectedValue: Long): Check = actualValue => + assert(actualValue === Some(expectedValue)) + + def atLeast(minimumValue: Long): Check = { actualValue => + assert(actualValue.isDefined) + assert(actualValue.get >= minimumValue) + } + } + + object PhysicalPlan { + type Check = SparkPlan => Unit + + val ANY: Check = _ => () // Ignore. + + def contains[T <: SparkPlan: ClassTag](implicit cls: ClassTag[T]): Check = { plan => + val existsSomeNodeOfTypeT = + AdaptiveSparkPlanHelper.existsWithSubqueries(plan)(_.getClass == cls.runtimeClass) + assert( + existsSomeNodeOfTypeT, + s"Expected a node ${cls.runtimeClass.getSimpleName}. Actual Plan:\n${plan.treeString}") + } + + def exists(pf: PartialFunction[SparkPlan, Boolean]): Check = { plan => + val existsMatchingNode = + AdaptiveSparkPlanHelper.existsWithSubqueries(plan)(pf.lift(_).getOrElse(false)) + assert( + existsMatchingNode, + s"Unexpected plan (check match function). Actual Plan:\n${plan.treeString}") + } + + def isAQE: Boolean = SQLConf.get.getConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED) + + def hasStageRetries: Boolean = spark.sparkContext.conf + .getOption(config.Tests.INJECT_SHUFFLE_FETCH_FAILURES.key).contains("true") + + def hasAQEReplans: Boolean = AQETestHelper.isForcedCancellationEnabled + } + + protected def testPhysicalPlanShape( + label: String, + setup: () => Unit = () => (), + extraSQLConfs: Map[String, String] = Map.empty, + sqlQuery: String, + executedPlanCheck: PhysicalPlan.Check, + metricValueCheck: MetricValue.Check + )(testTags: Tag*): Unit = { + for { + useAQE <- BOOLEAN_DOMAIN + stageRetries <- BOOLEAN_DOMAIN + aqeReplans <- if (useAQE) BOOLEAN_DOMAIN else Seq(false) + } test(s"$label - " + + s"useAQE=$useAQE, stageRetries=$stageRetries, aqeReplans=$aqeReplans", + testTags: _*) { + + // There is some special handling for df.cache() / df.persist() / df.localCheckpoint() tests. + val cachedPlanTest = label.startsWith("cache - ") + + withSQLConf( + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> useAQE.toString) { + setup() + withSQLConf(extraSQLConfs.toSeq: _*) { + val aqeRetryMetrics = if (aqeReplans) Seq(testSLAMetric) else Seq.empty + AQETestHelper.withForcedCancellation(aqeRetryMetrics: _*) { + withSparkContextConf( + config.Tests.INJECT_SHUFFLE_FETCH_FAILURES.key -> stageRetries.toString) { + val resultDf = spark.sql(sqlQuery) + val _ = resultDf.collect() + + // normal value of the metrics shall not work with retries or replans + if (!stageRetries && !aqeReplans) { + metricValueCheck(Some(testSLAMetric.value)) + } + // test LastRDDValue + metricValueCheck(testSLAMetric.lastAttemptValueForHighestRDDId()) + // test Dataset value + if (!cachedPlanTest) { + // SLAM.lastAttemptValueForDataset is undefined when SLAM is inside + // cached or checkpointed plan. + metricValueCheck(testSLAMetric.lastAttemptValueForDataset(resultDf)) + } + // test expected plan shape + val executedPlan = resultDf.queryExecution.executedPlan + executedPlanCheck(executedPlan) + val rddIdExec = testSLAMetric.getHighestRDDId + + // Repeated execution should not affect SLAM metric value + resultDf.collect() + // test LastRDDValue again + metricValueCheck(testSLAMetric.lastAttemptValueForHighestRDDId()) + // test Dataset value again + if (!cachedPlanTest) { + // SLAM.lastAttemptValueForDataset is undefined when SLAM is inside + // cached or checkpointed plan. + metricValueCheck(testSLAMetric.lastAttemptValueForDataset(resultDf)) + } + + // count() transformation creates a new Dataset. + // It should not affect the SLAM metric value of the first Dataset. + resultDf.count() + // test Dataset value again + if (!cachedPlanTest) { + // SLAM.lastAttemptValueForDataset is undefined when SLAM is inside + // cached or checkpointed plan. + metricValueCheck(testSLAMetric.lastAttemptValueForDataset(resultDf)) + } + // This should have created a new plan and executed new RDDs, + // unless it's a test of cached plan. + val rddIdExecCount = testSLAMetric.getHighestRDDId + if (cachedPlanTest) { + assert(rddIdExecCount === rddIdExec) + } else { + // count() creates a new plan with new RDDs. + assert(rddIdExecCount.get > rddIdExec.get) + } + } + } + } + } + } + } + + protected def testPlanShape( + label: String, + sqlQuery: String, + // Assert on the result of the test metric. + metricValueCheck: MetricValue.Check, + testTags: Tag* + ): Unit = { + testPhysicalPlanShape( + label = label, + sqlQuery = sqlQuery, + executedPlanCheck = PhysicalPlan.ANY, + metricValueCheck = metricValueCheck + )(testTags: _*) + } + + testPlanShape( + label = "simple plan", + sqlQuery = s"SELECT * FROM $TABLE_NAME WHERE increment_metric()", + metricValueCheck = MetricValue.exactly(NUM_RECORDS) + ) + + /* ******************** + * Various Subquery Plans + * ********************** */ + testPlanShape( + label = "subquery - IN", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME + | WHERE id IN ( + | SELECT low_cardinality_col + | FROM $TABLE_NAME + | WHERE increment_metric())""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS) + ) + + testPlanShape( + label = "subquery - IN - aggregation", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME + | WHERE id IN ( + | SELECT DISTINCT(low_cardinality_col) + | FROM $TABLE_NAME + | WHERE increment_metric())""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS) + ) + + testPlanShape( + label = "subquery - IN - TVF", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME + | WHERE id IN ( + | SELECT * + | FROM range(5) + | WHERE increment_metric())""".stripMargin, + metricValueCheck = MetricValue.exactly(5) + ) + + testPlanShape( + label = "subquery - IN - explode", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME + | WHERE id IN ( + | SELECT explode(array(low_cardinality_col, low_cardinality_col + 1)) + | FROM $TABLE_NAME + | WHERE increment_metric())""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS) + ) + + testPlanShape( + label = "subquery - IN - lateral view explode", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME + | WHERE id IN ( + | SELECT new_column + | FROM $TABLE_NAME LATERAL VIEW + | explode(array(low_cardinality_col, low_cardinality_col + 1)) AS new_column + | WHERE increment_metric())""".stripMargin, + metricValueCheck = MetricValue.exactly(2 * NUM_RECORDS) + ) + + testPlanShape( + label = "subquery - scalar", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME + | WHERE id == ( + | SELECT MAX(low_cardinality_col) + | FROM $TABLE_NAME + | WHERE increment_metric())""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS) + ) + + testPhysicalPlanShape( + label = "subquery - EXISTS", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME + | WHERE EXISTS ( + | SELECT low_cardinality_col + | FROM $TABLE_NAME + | WHERE increment_metric())""".stripMargin, + // This turns into a LIMIT query. + metricValueCheck = MetricValue.atLeast(1), + executedPlanCheck = PhysicalPlan.contains[CollectLimitExec] + )() + + testPhysicalPlanShape( + label = "subquery - EXISTS (correlated)", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME outer_table + | WHERE EXISTS ( + | SELECT low_cardinality_col + | FROM $TABLE_NAME inner_table + | WHERE increment_metric() + | AND inner_table.low_cardinality_col == outer_table.low_cardinality_col) + | """.stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS), + executedPlanCheck = PhysicalPlan.exists { + case _: BroadcastExchangeExec => true + case _: ShuffleExchangeExec => true + case _: ReusedExchangeExec => true + } + )() + + /* ***************************** + * Plans with different Exchanges + * ****************************** */ + + /* + * To cover: + * - ShuffleExchangeLike + * - ShuffleExchangeExec: covered by exchange - Shuffle + * - ReusedExchangeExec: covered by exchange - ReusedExchangeExec + * - BroadcastExchangeLike: + * - BroadcastExchangeExec: covered above by subquery - EXISTS (correlated)) + * - InMemoryTableScanLike (InMemoryTableScanExec): covered by exchange - InMemoryTableScanExec + */ + + testPhysicalPlanShape( + label = "exchange - Shuffle", + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME orig + | FULL OUTER JOIN ( + | SELECT * + | FROM $TABLE_NAME + | WHERE increment_metric() + | ) with_metric USING (id)""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS), + executedPlanCheck = PhysicalPlan.exists { + case _: ShuffleExchangeExec => true + // After forced AQE replans it may use ReusedExchange. + case _: ReusedExchangeExec if PhysicalPlan.hasAQEReplans => true + } + )() + + testPhysicalPlanShape( + label = "exchange - ReusedExchangeExec", + sqlQuery = + s"""WITH subquery_with_metric AS ( + | SELECT * + | FROM $TABLE_NAME + | WHERE increment_metric() + | ) + |SELECT * + | FROM subquery_with_metric a JOIN subquery_with_metric b USING (id)""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS), + executedPlanCheck = PhysicalPlan.contains[ReusedExchangeExec] + )() + + for (eager <- Seq("true", "false", "manual")) { + // SLAM metric in the top stage of cached query. + testPhysicalPlanShape( + label = s"cache - InMemoryTableScanExec - result stage - eager=$eager", + setup = () => { + spark.sql(s""" + |CREATE OR REPLACE TEMP VIEW table_with_metric AS ( + | SELECT low_cardinality_col + | FROM $TABLE_NAME + | WHERE increment_metric() + |)""".stripMargin) + if (eager == "true") { + spark.sql("CACHE TABLE table_with_metric") + } else { // false or manual + spark.sql("CACHE LAZY TABLE table_with_metric") + } + if (eager == "manual") { + spark.sql("select count(*) from table_with_metric").collect() + } + }, + sqlQuery = + s"""SELECT * + | FROM $TABLE_NAME + | WHERE id IN (SELECT * FROM table_with_metric)""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS), + executedPlanCheck = PhysicalPlan.contains[InMemoryTableScanExec] + )() + + // SLAM metric in the map stage of cached query. + testPhysicalPlanShape( + label = s"cache - InMemoryTableScanExec - map stage - eager=$eager", + setup = () => { + spark.sql(s""" + |CREATE OR REPLACE TEMP VIEW table_with_metric AS ( + | SELECT id, SUM(low_cardinality_col) + | FROM $TABLE_NAME + | WHERE increment_metric() + | GROUP BY id + |)""".stripMargin) + if (eager == "true") { + spark.sql("CACHE TABLE table_with_metric") + } else { // false or manual + spark.sql("CACHE LAZY TABLE table_with_metric") + } + if (eager == "manual") { + spark.sql("select count(*) from table_with_metric").collect() + } + }, + sqlQuery = + s"""SELECT * + | FROM table_with_metric""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS), + executedPlanCheck = PhysicalPlan.contains[InMemoryTableScanExec] + )() + + testPhysicalPlanShape( + label = s"cache - localCheckpoint - result stage - eager=$eager", + setup = () => { + val df = spark.sql(s""" + |SELECT low_cardinality_col + |FROM $TABLE_NAME + |WHERE increment_metric()""".stripMargin) + val cpEager = if (eager == "true") true else false + val cpDf = df.localCheckpoint(eager = cpEager) + if (eager == "manual") { + cpDf.count() + } + cpDf.createOrReplaceTempView("cp_table_with_metric") + }, + sqlQuery = + s"""SELECT * + | FROM cp_table_with_metric""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS), + executedPlanCheck = PhysicalPlan.contains[RDDScanExec] + )() + + testPhysicalPlanShape( + label = s"cache - localCheckpoint - map stage - eager=$eager", + setup = () => { + val df = spark.sql(s""" + |SELECT id, SUM(low_cardinality_col) + |FROM $TABLE_NAME + |WHERE increment_metric() + |GROUP BY id""".stripMargin) + val cpEager = if (eager == "true") true else false + val cpDf = df.localCheckpoint(eager = cpEager) + if (eager == "manual") { + cpDf.count() + } + cpDf.createOrReplaceTempView("cp_table_with_metric") + }, + sqlQuery = + s"""SELECT * + | FROM cp_table_with_metric""".stripMargin, + metricValueCheck = MetricValue.exactly(NUM_RECORDS), + executedPlanCheck = PhysicalPlan.contains[RDDScanExec] + )() + } +} + +object SQLLastAttemptMetricPlanShapesSuite { + val NUM_RECORDS: Long = 300 + val LOW_CARDINALITY: Int = 5 + val LARGE_CARDINALITY: Int = 111 + + val TABLE_NAME: String = "test_table" +} diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricUnitSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricUnitSuite.scala new file mode 100644 index 0000000000000..f3a696f81450e --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptMetricUnitSuite.scala @@ -0,0 +1,188 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.execution.metric + +import java.io.{ByteArrayInputStream, ByteArrayOutputStream, ObjectInputStream, ObjectOutputStream} +import java.util.Properties + +import org.mockito.Mockito.when +import org.scalatestplus.mockito.MockitoSugar.mock + +import org.apache.spark.{SharedSparkContext, SparkFunSuite} +import org.apache.spark.rdd.RDD +import org.apache.spark.scheduler.TaskInfo + +/** Tests internals of [[SQLLastAttemptMetric]]. */ +class SQLLastAttemptMetricUnitSuite extends SparkFunSuite with SharedSparkContext { + + // scalastyle:off classforname + private val sqlLastAttemptMetricClass = Class + .forName("org.apache.spark.sql.execution.metric.SQLLastAttemptMetric") + // scalastyle:on classforname + + private val lastAttemptInitializedField = + sqlLastAttemptMetricClass.getDeclaredField("lastAttemptAccumulatorInitialized") + + private val lastAttemptRddsMapField = + sqlLastAttemptMetricClass.getDeclaredField( + "org$apache$spark$util$LastAttemptAccumulator$$lastAttemptRddsMap") + + private val directDriverValueField = + sqlLastAttemptMetricClass.getDeclaredField( + "org$apache$spark$util$LastAttemptAccumulator$$lastAttemptDirectDriverValue") + + private val partialMergeValMethod = sqlLastAttemptMetricClass.getMethod("partialMergeVal") + + private val mockRdd = mock[RDD[_]] + private val mockTaskInfo = mock[TaskInfo] + private val mockProperties = new Properties + + // Set mock attempt for mock Task, TaskInfo and RDD + // that can be used with mergeLastAttempt. + // stageId and stageAttemptId are passed directly to mergeLastAttempt. + def setMockAttempt(rddId: Int, partitionId: Int): Unit = { + // reset to mock defaults + when(mockTaskInfo.attemptNumber).thenReturn(0) + when(mockRdd.scope).thenReturn(None) + when(mockRdd.getNumPartitions).thenReturn(5) + + when(mockRdd.id).thenReturn(rddId) + when(mockTaskInfo.partitionId).thenReturn(partitionId) + } + + override def beforeAll(): Unit = { + super.beforeAll() + lastAttemptInitializedField.setAccessible(true) + lastAttemptRddsMapField.setAccessible(true) + directDriverValueField.setAccessible(true) + partialMergeValMethod.setAccessible(true) + } + + override def afterAll(): Unit = { + lastAttemptInitializedField.setAccessible(false) + lastAttemptRddsMapField.setAccessible(false) + directDriverValueField.setAccessible(false) + partialMergeValMethod.setAccessible(false) + super.afterAll() + } + + test("serialization and deserialization") { + val slam = SQLLastAttemptMetrics.createMetric(sc, "test SLAM") + + assert(lastAttemptInitializedField.getBoolean(slam) === true) + assert(lastAttemptRddsMapField.get(slam) != null) + assert(directDriverValueField.get(slam) != null) + + // Serialize slam to ObjectOutputStream and deserialize it back. + val obs1 = new ByteArrayOutputStream() + val oos1 = new ObjectOutputStream(obs1) + oos1.writeObject(slam) + oos1.close() + val ois1 = new ObjectInputStream(new ByteArrayInputStream(obs1.toByteArray)) + val deser = ois1.readObject().asInstanceOf[SQLLastAttemptMetric] + + // serialized version should not be initialized + assert(lastAttemptInitializedField.getBoolean(deser) === false) + assert(lastAttemptRddsMapField.get(deser) == null) + assert(directDriverValueField.get(deser) == null) + + deser.set(42) + deser.add(7) + assert(deser.value === 49) + // these functions shouldn't be used on the deserialized metric, + // but assertions should be caught and None should be returned. + assert(deser.lastAttemptValueForHighestRDDId() === None) + assert(deser.lastAttemptValueForRDDId(1) === None) + assert(deser.lastAttemptValueForRDDIds(Seq(1, 2, 3)) === None) + assert(deser.lastAttemptValueForAllRDDs() === None) + // mergeLastAttempt shouldn't be used on the deserialized metric, + // but it should catch error and not fail. + deser.mergeLastAttempt(slam, null, null, 0, 0, null) + + // Serialize and deserialize again. + val obs2 = new ByteArrayOutputStream() + val oos2 = new ObjectOutputStream(obs2) + oos2.writeObject(deser) + oos2.close() + val ois2 = new ObjectInputStream(new ByteArrayInputStream(obs2.toByteArray)) + val reser = ois2.readObject().asInstanceOf[SQLLastAttemptMetric] + // Check that the value is brought back and can be used as partialMergeVal. + assert(reser.value === 49L) + assert(partialMergeValMethod.invoke(reser) === 49L) + } + + test("copy and mergeLastAttempt") { + val slam = SQLLastAttemptMetrics.createMetric(sc, "test SLAM") + + assert(lastAttemptInitializedField.getBoolean(slam) == true) + assert(lastAttemptRddsMapField.get(slam) != null) + assert(directDriverValueField.get(slam) != null) + + // copy should not initialize SLAM data. + val acc = slam.copy() + assert(lastAttemptInitializedField.getBoolean(acc) == false) + assert(lastAttemptRddsMapField.get(acc) == null) + assert(directDriverValueField.get(acc) == null) + // these functions shouldn't be used on the copy, + // but assertions should be caught and None should be returned. + assert(acc.lastAttemptValueForHighestRDDId() === None) + assert(acc.lastAttemptValueForRDDId(1) === None) + assert(acc.lastAttemptValueForRDDIds(Seq(1, 2, 3)) === None) + assert(acc.lastAttemptValueForAllRDDs() === None) + // mergeLastAttempt shouldn't be used on the copy, + // but it should catch error and not fail. + acc.mergeLastAttempt(slam, null, null, 0, 0, null) + + // Let's play with merging acc into slam. + setMockAttempt(rddId = 1, partitionId = 0) + acc.set(10) + slam.mergeLastAttempt(acc, mockRdd, mockTaskInfo, 10, 10, mockProperties) + assert(slam.lastAttemptValueForRDDId(1) === Some(10)) + + setMockAttempt(rddId = 1, partitionId = 1) + acc.set(10) // new partition id + slam.mergeLastAttempt(acc, mockRdd, mockTaskInfo, 10, 10, mockProperties) + assert(slam.lastAttemptValueForRDDId(1) === Some(20)) // 10 + 10, aggregated new partition id + + setMockAttempt(rddId = 1, partitionId = 1) + acc.set(7) // same partition id, older attempt. + slam.mergeLastAttempt(acc, mockRdd, mockTaskInfo, 10, 9, mockProperties) + assert(slam.lastAttemptValueForRDDId(1) === Some(20)) // no change + + setMockAttempt(rddId = 1, partitionId = 1) + acc.set(7) // same partition id, older stage. + slam.mergeLastAttempt(acc, mockRdd, mockTaskInfo, 9, 11, mockProperties) + assert(slam.lastAttemptValueForRDDId(1) === Some(20)) // no change + + setMockAttempt(rddId = 1, partitionId = 1) + acc.set(7) // same partition id, newer attempt. + slam.mergeLastAttempt(acc, mockRdd, mockTaskInfo, 10, 11, mockProperties) + assert(slam.lastAttemptValueForRDDId(1) === Some(17)) // 10 replaced with 7 + + setMockAttempt(rddId = 1, partitionId = 1) + acc.set(8) // same partition id, newer stage. + slam.mergeLastAttempt(acc, mockRdd, mockTaskInfo, 11, 1, mockProperties) + assert(slam.lastAttemptValueForRDDId(1) === Some(18)) // 7 replaced with 8 + + setMockAttempt(rddId = 2, partitionId = 2) + acc.set(42) // new RDD + slam.mergeLastAttempt(acc, mockRdd, mockTaskInfo, 1, 1, mockProperties) + assert(slam.lastAttemptValueForRDDId(1) === Some(18)) // no change for rddId=1 + assert(slam.lastAttemptValueForRDDId(2) === Some(42)) // new RDD added + assert(slam.lastAttemptValueForAllRDDs() === Some(60)) + } +} diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala index e8902ed6fb1a1..483d2a72637d1 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala @@ -25,14 +25,38 @@ import org.apache.spark.TestUtils import org.apache.spark.scheduler.{SparkListener, SparkListenerTaskEnd} import org.apache.spark.sql.{DataFrame, QueryTest} import org.apache.spark.sql.catalyst.TableIdentifier +import org.apache.spark.sql.catalyst.expressions.Expression import org.apache.spark.sql.execution.{SparkPlan, SparkPlanInfo} import org.apache.spark.sql.execution.ui.{SparkPlanGraph, SQLAppStatusStore} +import org.apache.spark.sql.functions.udf import org.apache.spark.sql.internal.SQLConf.WHOLESTAGE_CODEGEN_ENABLED trait SQLMetricsTestUtils extends QueryTest { import testImplicits._ + protected val BOOLEAN_DOMAIN: Seq[Boolean] = Seq(true, false) + + /** + * @return An `Expression` that increments a SQL metric and + * evaluates to true. Can be used in a filter. + */ + protected def incrementMetric( + metric: SQLMetric): Expression = { + udf { () => + { metric += 1; true } + }.asNondeterministic().apply().expr + } + + /** @return An `Expression` to increment multiple SQL metrics */ + protected def incrementMetrics(metrics: Seq[SQLMetric]): Expression = { + metrics.map(incrementMetric(_)).fold( + org.apache.spark.sql.catalyst.expressions.Literal(true): Expression) { + (acc, incrMetric) => + org.apache.spark.sql.catalyst.expressions.And(acc, incrMetric) + } + } + protected def currentExecutionIds(): Set[Long] = { spark.sparkContext.listenerBus.waitUntilEmpty(10000) statusStore.executionsList().map(_.executionId).toSet diff --git a/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions b/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions index 4563e81d14064..2aa6cb885ca31 100644 --- a/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions +++ b/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions @@ -1173,6 +1173,7 @@ spark.taskMetrics.trackUpdatedBlockStatuses spark.test.noStageRetry spark.testing spark.testing.dynamicAllocation.schedule.enabled +spark.testing.injectShuffleFetchFailures spark.testing.memory spark.testing.nCoresPerExecutor spark.testing.nExecutorsPerHost From 0b1bcbd4eec686e041e84e7cea3df4ad93c94749 Mon Sep 17 00:00:00 2001 From: Ziya Mukhtarov Date: Tue, 28 Apr 2026 07:41:05 -0700 Subject: [PATCH 004/286] [SPARK-56551][SQL][FOLLOW-UP] Add DSv2 DML tests with always true/false filters ### What changes were proposed in this pull request? Add more DSv2 tests with always true/false conditions for DML commands. ### Why are the changes needed? N/A. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? It is adding more tests. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 Closes #55516 from ZiyaZa/dsv2-dml-more-tests. Authored-by: Ziya Mukhtarov Signed-off-by: Anton Okolnychyi --- .../connector/DeleteFromTableSuiteBase.scala | 60 +++++++++++ .../connector/MergeIntoTableSuiteBase.scala | 99 +++++++++++++++++++ .../sql/connector/UpdateTableSuiteBase.scala | 32 ++++++ 3 files changed, 191 insertions(+) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeleteFromTableSuiteBase.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeleteFromTableSuiteBase.scala index 2682487e51ba0..adc88f5a54a07 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeleteFromTableSuiteBase.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeleteFromTableSuiteBase.scala @@ -179,6 +179,66 @@ abstract class DeleteFromTableSuiteBase extends RowLevelOperationSuiteBase { checkDeleteMetrics(numDeletedRows = 0, numCopiedRows = 0) } + test("delete with literal false condition") { + createAndInitTable("pk INT NOT NULL, id INT, dep STRING", + """{ "pk": 1, "id": 1, "dep": "hr" } + |{ "pk": 2, "id": 2, "dep": "software" } + |{ "pk": 3, "id": 3, "dep": "hr" } + |""".stripMargin) + + sql(s"DELETE FROM $tableNameAsString WHERE false") + + checkAnswer( + sql(s"SELECT * FROM $tableNameAsString"), + Row(1, 1, "hr") :: Row(2, 2, "software") :: Row(3, 3, "hr") :: Nil) + + checkDeleteMetrics(numDeletedRows = 0, numCopiedRows = 0) + } + + test("delete with literal true condition") { + createAndInitTable("pk INT NOT NULL, id INT, dep STRING", + """{ "pk": 1, "id": 1, "dep": "hr" } + |{ "pk": 2, "id": 2, "dep": "software" } + |{ "pk": 3, "id": 3, "dep": "hr" } + |""".stripMargin) + + sql(s"DELETE FROM $tableNameAsString WHERE true") + + checkAnswer(sql(s"SELECT * FROM $tableNameAsString"), Nil) + } + + test("delete with NULL equality on VOID column") { + createAndInitTable("pk INT NOT NULL, v VOID, dep STRING", + """{ "pk": 1, "v": null, "dep": "hr" } + |{ "pk": 2, "v": null, "dep": "software" } + |{ "pk": 3, "v": null, "dep": "hr" } + |""".stripMargin) + + sql(s"DELETE FROM $tableNameAsString WHERE v = NULL") + + checkAnswer( + sql(s"SELECT pk, dep FROM $tableNameAsString"), + Row(1, "hr") :: Row(2, "software") :: Row(3, "hr") :: Nil) + + checkDeleteMetrics(numDeletedRows = 0, numCopiedRows = 0) + } + + test("delete with NULL condition on non-null column") { + createAndInitTable("pk INT NOT NULL, id INT, dep STRING", + """{ "pk": 1, "id": 1, "dep": "hr" } + |{ "pk": 2, "id": 2, "dep": "software" } + |{ "pk": 3, "id": 3, "dep": "hr" } + |""".stripMargin) + + sql(s"DELETE FROM $tableNameAsString WHERE pk = NULL") + + checkAnswer( + sql(s"SELECT * FROM $tableNameAsString"), + Row(1, 1, "hr") :: Row(2, 2, "software") :: Row(3, 3, "hr") :: Nil) + + checkDeleteMetrics(numDeletedRows = 0, numCopiedRows = 0) + } + test("delete with basic filters") { createAndInitTable("pk INT NOT NULL, id INT, dep STRING", """{ "pk": 1, "id": 1, "dep": "hr" } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala index 7c0e503705c7e..069781e40d8c2 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala @@ -249,6 +249,105 @@ abstract class MergeIntoTableSuiteBase extends RowLevelOperationSuiteBase } } + test("merge with literal false ON condition") { + withTempView("source") { + createAndInitTable("pk INT NOT NULL, salary INT, dep STRING", + """{ "pk": 1, "salary": 100, "dep": "hr" } + |{ "pk": 2, "salary": 200, "dep": "hardware" } + |""".stripMargin) + + Seq((1, 100, "hr"), (3, 300, "finance")) + .toDF("pk", "salary", "dep").createOrReplaceTempView("source") + + sql( + s"""MERGE INTO $tableNameAsString t + |USING source s + |ON false + |WHEN MATCHED THEN UPDATE SET t.salary = -1 + |""".stripMargin) + + checkAnswer( + sql(s"SELECT * FROM $tableNameAsString"), + Seq(Row(1, 100, "hr"), Row(2, 200, "hardware"))) + + val summary = getMergeSummary() + assert(summary.numTargetRowsUpdated === 0L) + assert(summary.numTargetRowsDeleted === 0L) + assert(summary.numTargetRowsInserted === 0L) + assert(summary.numTargetRowsMatchedUpdated === 0L) + assert(summary.numTargetRowsMatchedDeleted === 0L) + assert(summary.numTargetRowsNotMatchedBySourceUpdated === 0L) + assert(summary.numTargetRowsNotMatchedBySourceDeleted === 0L) + assert(summary.numTargetRowsCopied === 0L) + } + } + + test("merge with literal true ON condition") { + withTempView("source") { + createAndInitTable("pk INT NOT NULL, salary INT, dep STRING", + """{ "pk": 1, "salary": 100, "dep": "hr" } + |{ "pk": 2, "salary": 200, "dep": "hardware" } + |""".stripMargin) + + Seq((99, 999, "finance")) + .toDF("pk", "salary", "dep").createOrReplaceTempView("source") + + sql( + s"""MERGE INTO $tableNameAsString t + |USING source s + |ON true + |WHEN MATCHED THEN UPDATE SET t.salary = -1 + |""".stripMargin) + + checkAnswer( + sql(s"SELECT * FROM $tableNameAsString"), + Seq(Row(1, -1, "hr"), Row(2, -1, "hardware"))) + + val summary = getMergeSummary() + assert(summary.numTargetRowsUpdated === 2L) + assert(summary.numTargetRowsDeleted === 0L) + assert(summary.numTargetRowsInserted === 0L) + assert(summary.numTargetRowsMatchedUpdated === 2L) + assert(summary.numTargetRowsMatchedDeleted === 0L) + assert(summary.numTargetRowsNotMatchedBySourceUpdated === 0L) + assert(summary.numTargetRowsNotMatchedBySourceDeleted === 0L) + assert(summary.numTargetRowsCopied === 0L) + } + } + + test("merge with statically empty source and only MATCHED clauses") { + withTempView("source") { + createAndInitTable("pk INT NOT NULL, salary INT, dep STRING", + """{ "pk": 1, "salary": 100, "dep": "hr" } + |{ "pk": 2, "salary": 200, "dep": "hardware" } + |""".stripMargin) + + Seq.empty[(Int, Int, String)].toDF("pk", "salary", "dep") + .createOrReplaceTempView("source") + + sql( + s"""MERGE INTO $tableNameAsString t + |USING source s + |ON t.pk = s.pk + |WHEN MATCHED THEN UPDATE SET t.salary = -1 + |""".stripMargin) + + checkAnswer( + sql(s"SELECT * FROM $tableNameAsString"), + Seq(Row(1, 100, "hr"), Row(2, 200, "hardware"))) + + val summary = getMergeSummary() + assert(summary.numTargetRowsUpdated === 0L) + assert(summary.numTargetRowsDeleted === 0L) + assert(summary.numTargetRowsInserted === 0L) + assert(summary.numTargetRowsMatchedUpdated === 0L) + assert(summary.numTargetRowsMatchedDeleted === 0L) + assert(summary.numTargetRowsNotMatchedBySourceUpdated === 0L) + assert(summary.numTargetRowsNotMatchedBySourceDeleted === 0L) + assert(summary.numTargetRowsCopied === 0L) + } + } + test("merge into with conditional WHEN MATCHED clause (update)") { withTempView("source") { createAndInitTable("pk INT NOT NULL, salary INT, dep STRING", diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala index d3a6b61a61b9f..d32a1e5c7f561 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala @@ -222,6 +222,38 @@ abstract class UpdateTableSuiteBase extends RowLevelOperationSuiteBase { checkUpdateMetrics(numUpdatedRows = 0, numCopiedRows = 0) } + test("update with literal false condition") { + createAndInitTable("pk INT NOT NULL, salary INT, dep STRING", + """{ "pk": 1, "salary": 100, "dep": "hr" } + |{ "pk": 2, "salary": 200, "dep": "hardware" } + |{ "pk": 3, "salary": null, "dep": "hr" } + |""".stripMargin) + + sql(s"UPDATE $tableNameAsString SET salary = -1 WHERE false") + + checkAnswer( + sql(s"SELECT * FROM $tableNameAsString"), + Row(1, 100, "hr") :: Row(2, 200, "hardware") :: Row(3, null, "hr") :: Nil) + + checkUpdateMetrics(numUpdatedRows = 0, numCopiedRows = 0) + } + + test("update with literal true condition") { + createAndInitTable("pk INT NOT NULL, salary INT, dep STRING", + """{ "pk": 1, "salary": 100, "dep": "hr" } + |{ "pk": 2, "salary": 200, "dep": "hardware" } + |{ "pk": 3, "salary": null, "dep": "hr" } + |""".stripMargin) + + sql(s"UPDATE $tableNameAsString SET salary = -1 WHERE true") + + checkAnswer( + sql(s"SELECT * FROM $tableNameAsString"), + Row(1, -1, "hr") :: Row(2, -1, "hardware") :: Row(3, -1, "hr") :: Nil) + + checkUpdateMetrics(numUpdatedRows = 3, numCopiedRows = 0) + } + test("update without condition") { createAndInitTable("pk INT NOT NULL, salary INT, dep STRING", """{ "pk": 1, "salary": 100, "dep": "hr" } From d3bf35390c76d062ce26047bc0d2ca3e37083d95 Mon Sep 17 00:00:00 2001 From: Serge Rielau Date: Tue, 28 Apr 2026 09:44:27 -0700 Subject: [PATCH 005/286] [SPARK-56605][SQL] Wire resolution engine to use SQL PATH for table, function, and variable lookup ### What changes were proposed in this pull request? Switch the resolution engine from the legacy single-schema `resolutionSearchPath` to `sqlResolutionPathEntries` on `CatalogManager`, so that `SET PATH` actually affects how unqualified table names, function names, and variables are resolved. **CatalogManager** (`CatalogManager.scala`): - `sqlResolutionPathEntries`: ordered path entries for resolving unqualified names. When PATH is enabled and set, uses the stored session path; otherwise falls back to `defaultPathOrder` (legacy). - `sessionScopeUnqualifiedAllowed`: gates unqualified variable access when `system.session` is not on the PATH. **Relation resolution** (`RelationResolution.scala`): - `relationResolutionEntries` / `relationResolutionSteps` replace `relationResolutionSearchPath`. - `PersistentCatalogStep` carries the catalog/namespace prefix so each path entry qualifies the object name under that entry. **Function resolution** (`FunctionResolution.scala`): - `sqlResolutionPathEntriesForAnalysis` provides candidates for unqualified function names. - Procedure resolution (`resolveProcedure`) moved here from `Analyzer`. **Error context** (`CheckAnalysis.scala`): - `catalogPathForError` now consults `AnalysisContext.catalogAndNamespace` when inside a view body, so error messages report the view's defining catalog/namespace. - `UnresolvedTableOrViewSearchPathMode` enum controls DDL vs query-like error paths. - Uses `CatalogManager` constants instead of string literals. - `RelationChanges` errors now include the search path. **Variable resolution** (`VariableResolution.scala` + callers): - `allowUnqualifiedSessionTempVariableLookup` gates unqualified variable access when `system.session` is not on the PATH. **Analyzer** (`Analyzer.scala`): - `sessionConf` constructor parameter for isolated analysis (e.g. Connect). - `resolutionConf` for path-based resolution. - `AnalysisContext.resolutionPathEntries` field (initially `None`; frozen path wiring comes in follow-up PR). **Single-pass resolver**: `Resolver`, `HybridAnalyzer`, `NameScope`, `ResolverGuard` aligned with new resolution signatures. Frozen path analysis for views/SQL functions (using stored path during analysis) comes in a follow-up PR. This PR only wires the resolution engine to use the live session path. ### Why are the changes needed? `SET PATH` (merged in [SPARK-56501](https://github.com/apache/spark/pull/55364)) stores the session path but the resolvers still used the legacy single-schema path. This PR completes the wiring so PATH actually affects resolution. Part of [SPARK-54810](https://issues.apache.org/jira/browse/SPARK-54810). ### Does this PR introduce _any_ user-facing change? Yes. With `spark.sql.path.enabled = true` and `SET PATH`, unqualified table names, function names, and variable references now resolve according to the stored path order. ### How was this patch tested? CI. `ProtoToParsedPlanTestSuite` updated with analyzer isolation conf. Connect `.explain` golden files updated. `PlanResolutionSuite`, `NameScopeSuite`, `TimezoneAwareExpressionResolverSuite` updated for new constructor signatures. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.6 Closes #55523 from srielau/SPARK-56605-resolution. Authored-by: Serge Rielau Signed-off-by: Gengliang Wang --- .../sql/catalyst/analysis/Analyzer.scala | 62 +++++---- .../sql/catalyst/analysis/CheckAnalysis.scala | 62 +++++---- .../analysis/ColumnResolutionHelper.scala | 3 +- .../analysis/FunctionResolution.scala | 123 ++++++++++++++++-- .../analysis/RelationResolution.scala | 77 +++++++---- .../catalyst/analysis/ResolveCatalogs.scala | 6 + .../analysis/ResolveFetchCursor.scala | 3 +- .../analysis/ResolveSetVariable.scala | 3 +- .../analysis/VariableResolution.scala | 19 ++- .../analysis/resolver/NameScope.scala | 4 +- .../catalyst/analysis/resolver/Resolver.scala | 20 ++- .../analysis/resolver/ResolverGuard.scala | 7 +- .../catalyst/analysis/v2ResolutionPlans.scala | 24 +++- .../sql/catalyst/parser/AstBuilder.scala | 8 +- .../connector/catalog/CatalogManager.scala | 37 ++++++ .../analysis/TableLookupCacheSuite.scala | 9 ++ ...TimezoneAwareExpressionResolverSuite.scala | 4 +- .../explain-results/read_changes.explain | 4 +- .../read_changes_with_options.explain | 4 +- .../explain-results/read_table.explain | 4 +- ...streaming_changes_API_with_options.explain | 4 +- .../streaming_table_API_with_options.explain | 4 +- .../query-tests/explain-results/table.explain | 4 +- .../table_API_with_options.explain | 4 +- .../connect/ProtoToParsedPlanTestSuite.scala | 26 +++- .../apache/spark/sql/classic/Catalog.scala | 6 +- .../spark/sql/execution/SparkSqlParser.scala | 9 +- .../sql/execution/command/SetCommand.scala | 4 +- .../internal/BaseSessionStateBuilder.scala | 2 +- .../sql-tests/results/describe.sql.out | 2 +- .../org/apache/spark/sql/SetPathSuite.scala | 80 +++++++++++- .../analysis/resolver/NameScopeSuite.scala | 1 + .../spark/sql/connector/ProcedureSuite.scala | 12 ++ .../command/AlignAssignmentsSuiteBase.scala | 7 + .../execution/command/DDLParserSuite.scala | 2 +- .../command/DescribeTableParserSuite.scala | 33 +++-- .../command/PlanResolutionSuite.scala | 17 +++ 37 files changed, 545 insertions(+), 155 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala index 0277b664ff904..323a7db9c7ad7 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala @@ -25,7 +25,7 @@ import scala.collection.mutable.ArrayBuffer import scala.jdk.CollectionConverters._ import scala.util.{Failure, Random, Success, Try} -import org.apache.spark.{SparkException, SparkThrowable, SparkUnsupportedOperationException} +import org.apache.spark.{SparkException, SparkUnsupportedOperationException} import org.apache.spark.internal.config.ConfigBindingPolicy import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst._ @@ -139,10 +139,16 @@ object FakeV2SessionCatalog extends TableCatalog with FunctionCatalog with Suppo * even if a temp view `t` has been created. * @param outerPlan The query plan from the outer query that can be used to resolve star * expressions in a subquery. + * @param resolutionPathEntries When resolving a view body, the ordered path for unqualified + * relation names. Stays [[None]] in this PR; population from the + * frozen path stored in view metadata is wired in a follow-up. + * Outside views: compute from session + * [[CatalogManager.sqlResolutionPathEntries]]. */ case class AnalysisContext( isDefault: Boolean = false, catalogAndNamespace: Seq[String] = Nil, + resolutionPathEntries: Option[Seq[Seq[String]]] = None, nestedViewDepth: Int = 0, maxNestedViewDepth: Int = -1, relationCache: mutable.Map[(Seq[String], Option[TimeTravelSpec]), LogicalPlan] = @@ -198,7 +204,6 @@ object AnalysisContext { def withAnalysisContext[A](viewDesc: CatalogTable)(f: => A): A = { val originContext = value.get() val maxNestedViewDepth = if (originContext.maxNestedViewDepth == -1) { - // Here we start to resolve views, get `maxNestedViewDepth` from configs. SQLConf.get.maxNestedViewDepth } else { originContext.maxNestedViewDepth @@ -290,12 +295,14 @@ object Analyzer { */ class Analyzer( override val catalogManager: CatalogManager, - private[sql] val sharedRelationCache: RelationCache = RelationCache.empty) + private[sql] val sharedRelationCache: RelationCache = RelationCache.empty, + private[sql] val sessionConf: Option[SQLConf] = None) extends RuleExecutor[LogicalPlan] with CheckAnalysis with AliasHelper with SQLConfHelper with ColumnResolutionHelper { private val v1SessionCatalog: SessionCatalog = catalogManager.v1SessionCatalog - private val relationResolution = new RelationResolution(catalogManager, sharedRelationCache) + private val relationResolution = + new RelationResolution(catalogManager, sharedRelationCache) private val functionResolution = new FunctionResolution(catalogManager, relationResolution) override protected def validatePlanChanges( @@ -317,11 +324,16 @@ class Analyzer( if (plan.analyzed) { plan } else { + def runAnalysis(): LogicalPlan = HybridAnalyzer.fromLegacyAnalyzer( + legacyAnalyzer = this, tracker = tracker).apply(plan) if (AnalysisContext.get.isDefault) { AnalysisContext.reset() try { AnalysisHelper.markInAnalyzer { - HybridAnalyzer.fromLegacyAnalyzer(legacyAnalyzer = this, tracker = tracker).apply(plan) + sessionConf match { + case Some(c) => SQLConf.withExistingConf(c) { runAnalysis() } + case None => runAnalysis() + } } } finally { AnalysisContext.reset() @@ -329,7 +341,10 @@ class Analyzer( } else { AnalysisContext.withNewAnalysisContext { AnalysisHelper.markInAnalyzer { - HybridAnalyzer.fromLegacyAnalyzer(legacyAnalyzer = this, tracker = tracker).apply(plan) + sessionConf match { + case Some(c) => SQLConf.withExistingConf(c) { runAnalysis() } + case None => runAnalysis() + } } } } @@ -342,7 +357,13 @@ class Analyzer( } } - private def executeSameContext(plan: LogicalPlan): LogicalPlan = super.execute(plan) + private def executeSameContext(plan: LogicalPlan): LogicalPlan = sessionConf match { + // Respect explicit nested SQLConf overrides (e.g. persisted SQL UDF/view configs). + // Otherwise, run analysis with the captured session conf for analyzer isolation. + case Some(c) if SQLConf.get ne c => super.execute(plan) + case Some(c) => SQLConf.withExistingConf(c) { super.execute(plan) } + case None => super.execute(plan) + } def resolver: Resolver = conf.resolver @@ -982,7 +1003,8 @@ class Analyzer( // This is done by keeping the catalog and namespace in `AnalysisContext`, and analyzer will // look at `AnalysisContext.catalogAndNamespace` when resolving relations with single-part name. // If `AnalysisContext.catalogAndNamespace` is non-empty, analyzer will expand single-part names - // with it, instead of current catalog and namespace. + // with it, instead of current catalog and namespace. Unqualified relation PATH will be + // snapshotted in `AnalysisContext.resolutionPathEntries` in a follow-up PR. private def resolveViews( plan: LogicalPlan, options: CaseInsensitiveStringMap): LogicalPlan = plan match { @@ -1091,7 +1113,7 @@ class Analyzer( case other => other }.getOrElse(u) - case u @ UnresolvedTableOrView(identifier, cmd, allowTempView) => + case u @ UnresolvedTableOrView(identifier, cmd, allowTempView, _) => lookupTableOrView(identifier).map { case _: ResolvedTempView if !allowTempView => throw QueryCompilationErrors.expectPermanentViewNotTempViewError( @@ -2133,10 +2155,8 @@ class Analyzer( throw QueryCompilationErrors.notAScalarFunctionError(nameParts.mkString("."), f) case FunctionType.NotFound => - val catalogPath = - catalogManager.currentCatalog.name +: catalogManager.currentNamespace - val searchPath = SQLConf.get.resolutionSearchPath(catalogPath.toSeq) - .map(_.quoted) + val searchPath = + functionResolution.sqlResolutionPathEntriesForAnalysis.map(_.quoted) throw QueryCompilationErrors.unresolvedRoutineError( nameParts, searchPath, @@ -2333,20 +2353,8 @@ class Analyzer( object ResolveProcedures extends Rule[LogicalPlan] { def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning( _.containsPattern(UNRESOLVED_PROCEDURE), ruleId) { - case UnresolvedProcedure(CatalogAndIdentifier(catalog, ident)) => - val procedureCatalog = catalog.asProcedureCatalog - val procedure = load(procedureCatalog, ident) - ResolvedProcedure(procedureCatalog, ident, procedure) - } - - private def load(catalog: ProcedureCatalog, ident: Identifier): UnboundProcedure = { - try { - catalog.loadProcedure(ident) - } catch { - case e: Exception if !e.isInstanceOf[SparkThrowable] => - val nameParts = catalog.name +: ident.asMultipartIdentifier - throw QueryCompilationErrors.failedToLoadRoutineError(nameParts, e) - } + case u: UnresolvedProcedure => + functionResolution.resolveProcedure(u) } } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala index 9d48955cbc71e..b923d442e6d98 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala @@ -16,8 +16,6 @@ */ package org.apache.spark.sql.catalyst.analysis -import java.util.Locale - import scala.collection.mutable import org.apache.spark.{SparkException, SparkThrowable} @@ -45,6 +43,8 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString protected def isView(nameParts: Seq[String]): Boolean + protected def conf: org.apache.spark.sql.internal.SQLConf + import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ /** @@ -73,20 +73,31 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString * Contains system.session and the current catalog namespace only. Not from SQLConf. */ private def ddlSearchPathForError(catalogPath: Seq[String]): Seq[String] = { - Seq(toSQLId(Seq("system", "session")), toSQLId(catalogPath)) + val sessionPath = Seq( + CatalogManager.SYSTEM_CATALOG_NAME, + CatalogManager.SESSION_NAMESPACE) + Seq(toSQLId(sessionPath), toSQLId(catalogPath)) } /** - * `SQLConf.resolutionSearchPath` entries formatted with [[toSQLId]] for TABLE_OR_VIEW_NOT_FOUND. - * Same ordering as relation resolution and routine resolution search paths. + * Formats [[CatalogManager.sqlResolutionPathEntries]] with [[toSQLId]] + * for TABLE_OR_VIEW_NOT_FOUND error messages. */ private def fullSearchPathForError(catalogPath: Seq[String]): Seq[String] = { - SQLConf.get.resolutionSearchPath(catalogPath).map(toSQLId) + val catalog = catalogPath.head + val ns = catalogPath.tail.toSeq + catalogManager.sqlResolutionPathEntries(catalog, ns).map(toSQLId) } - /** Current catalog name and namespace as a path, used when computing search path for errors. */ - private def catalogPathForError: Seq[String] = { - (currentCatalog.name +: catalogManager.currentNamespace).toSeq + /** + * Catalog + namespace path for error messages. When resolving inside a view body, + * uses the view's defining catalog/namespace from AnalysisContext so the error + * reflects where the view was trying to resolve. + */ + protected final def catalogPathForError: Seq[String] = { + val ctx = AnalysisContext.get.catalogAndNamespace + if (ctx.nonEmpty) ctx + else (currentCatalog.name +: catalogManager.currentNamespace).toSeq } /** @@ -94,13 +105,15 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString * (e.g. DROP TEMPORARY VIEW). Contains system.session only. */ private def tempViewOnlySearchPathForError(): Seq[String] = { - Seq(toSQLId(Seq("system", "session"))) + Seq(toSQLId(Seq( + CatalogManager.SYSTEM_CATALOG_NAME, + CatalogManager.SESSION_NAMESPACE))) } /** * Search path for TABLE_OR_VIEW_NOT_FOUND on unresolved relations in SELECT/DML/INSERT/time * travel. Three-part `system.session.name` resolves only to session temp views, so only that - * scope is listed. Other names use [[fullSearchPathForError]] (resolutionSearchPath order). + * scope is listed. Other names use [[fullSearchPathForError]] (sqlResolutionPathEntries order). */ private def searchPathForUnresolvedRelation(multipartIdentifier: Seq[String]): Seq[String] = { if (CatalogManager.isFullyQualifiedSystemSessionViewName(multipartIdentifier)) { @@ -381,17 +394,15 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString case u: UnresolvedTableOrView => val catalogPath = catalogPathForError - val searchPath = if (u.commandName.toUpperCase(Locale.ROOT).contains("TEMPORARY VIEW")) { - tempViewOnlySearchPathForError() - } else if (u.commandName.toUpperCase(Locale.ROOT).startsWith("DESCRIBE") || - u.commandName.toUpperCase(Locale.ROOT).startsWith("DESC ")) { - if (CatalogManager.isFullyQualifiedSystemSessionViewName(u.multipartIdentifier)) { - tempViewOnlySearchPathForError() - } else { - fullSearchPathForError(catalogPath) - } - } else { - ddlSearchPathForError(catalogPath) + val searchPath = u.tableNotFoundSearchPathMode match { + case UnresolvedTableOrViewSearchPathMode.QueryLike => + if (CatalogManager.isFullyQualifiedSystemSessionViewName(u.multipartIdentifier)) { + tempViewOnlySearchPathForError() + } else { + fullSearchPathForError(catalogPath) + } + case UnresolvedTableOrViewSearchPathMode.Ddl => + ddlSearchPathForError(catalogPath) } u.tableNotFound(u.multipartIdentifier, searchPath) @@ -401,8 +412,7 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString searchPathForUnresolvedRelation(u.multipartIdentifier)) case u: UnresolvedFunctionName => - val searchPath = - SQLConf.get.resolutionSearchPath(catalogPathForError).map(_.quoted) + val searchPath = fullSearchPathForError(catalogPathForError) throw QueryCompilationErrors.unresolvedRoutineError( u.multipartIdentifier, searchPath, @@ -595,7 +605,9 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString searchPathForUnresolvedRelation(u.multipartIdentifier)) case RelationChanges(u: UnresolvedRelation, _) => - u.tableNotFound(u.multipartIdentifier) + u.tableNotFound( + u.multipartIdentifier, + searchPathForUnresolvedRelation(u.multipartIdentifier)) case etw: EventTimeWatermark => etw.eventTime.dataType match { diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala index 93d71642ac9fd..8d8b8462a95be 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala @@ -258,7 +258,8 @@ trait ColumnResolutionHelper extends Logging with DataTypeErrorsBase { // Resolves `UnresolvedAttribute` to its value. protected def resolveVariables(e: Expression): Expression = { - val variableResolution = new VariableResolution(catalogManager.tempVariableManager) + val variableResolution = + new VariableResolution(catalogManager.tempVariableManager, catalogManager) def resolve(nameParts: Seq[String]): Option[Expression] = { variableResolution.resolveMultipartName( diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala index 4f3428cc69739..8f8c77f38feac 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala @@ -19,9 +19,12 @@ package org.apache.spark.sql.catalyst.analysis import java.util.concurrent.atomic.AtomicBoolean +import scala.util.control.NonFatal + +import org.apache.spark.SparkThrowable import org.apache.spark.internal.Logging import org.apache.spark.sql.AnalysisException -import org.apache.spark.sql.catalyst.FunctionIdentifier +import org.apache.spark.sql.catalyst.{FunctionIdentifier, SQLConfHelper} import org.apache.spark.sql.catalyst.analysis.FunctionRegistry import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.catalyst.expressions.aggregate._ @@ -32,7 +35,8 @@ import org.apache.spark.sql.connector.catalog.{ CatalogPlugin, CatalogV2Util, Identifier, - LookupCatalog + LookupCatalog, + ProcedureCatalog } /** @@ -56,14 +60,13 @@ import org.apache.spark.sql.connector.catalog.functions.{ UnboundFunction } import org.apache.spark.sql.errors.{DataTypeErrorsBase, QueryCompilationErrors} -import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.internal.connector.V1Function import org.apache.spark.sql.types._ class FunctionResolution( override val catalogManager: CatalogManager, relationResolution: RelationResolution) - extends DataTypeErrorsBase with LookupCatalog with Logging { + extends DataTypeErrorsBase with LookupCatalog with SQLConfHelper with Logging { private val v1SessionCatalog = catalogManager.v1SessionCatalog private val trimWarningEnabled = new AtomicBoolean(true) @@ -83,23 +86,41 @@ class FunctionResolution( /** * Produces the ordered list of candidate names for resolution. Expansion happens in two cases: * - * 1. Single-part names: expanded via the search path, where each search path entry is - * fully qualified so appending the name produces fully qualified candidates. + * 1. Single-part names: expanded via [[CatalogManager.sqlResolutionPathEntries]] (same list as + * relation resolution), where each path entry is fully qualified so appending the name + * produces fully qualified candidates. * 2. `builtin.name` or `session.name`: prepending `system` creates a fully qualified * system catalog candidate. The original 2-part name is also kept as a persistent * catalog candidate (qualified downstream). Order is controlled by * the `persistentCatalogFirst` config. * * All other multi-part names are returned as-is for downstream resolution. + * + * When [[AnalysisContext.resolutionPathEntries]] is set (view or SQL function / table function + * body with a pinned path, with [[SQLConf.PATH_ENABLED]] true), that frozen list is used + * directly, matching [[RelationResolution.relationResolutionEntries]] so routine order stays + * aligned with relation order. */ + private[analysis] def sqlResolutionPathEntriesForAnalysis: Seq[Seq[String]] = { + AnalysisContext.get.resolutionPathEntries match { + case Some(entries) if conf.pathEnabled => entries + case _ => + val pathDefault = currentCatalogPath + catalogManager.sqlResolutionPathEntries( + pathDefault.head, + pathDefault.tail.toSeq, + catalogManager.currentCatalog.name, + catalogManager.currentNamespace.toSeq) + } + } + private def resolutionCandidates(nameParts: Seq[String]): Seq[Seq[String]] = { if (nameParts.size == 1) { - val searchPath = SQLConf.get.resolutionSearchPath(currentCatalogPath) - searchPath.map(_ ++ nameParts) + sqlResolutionPathEntriesForAnalysis.map(_ ++ nameParts) } else if (nameParts.size == 2 && FunctionResolution.sessionNamespaceKind(nameParts).isDefined) { val systemCandidate = CatalogManager.SYSTEM_CATALOG_NAME +: nameParts - if (SQLConf.get.prioritizeSystemCatalog) { + if (conf.prioritizeSystemCatalog) { Seq(systemCandidate, nameParts) } else { Seq(nameParts, systemCandidate) @@ -174,9 +195,10 @@ class FunctionResolution( case None => } } - val searchPath = SQLConf.get.resolutionSearchPath(currentCatalogPath) throw QueryCompilationErrors.unresolvedRoutineError( - unresolvedFunc.nameParts, searchPath.map(toSQLId), unresolvedFunc.origin) + unresolvedFunc.nameParts, + sqlResolutionPathEntriesForAnalysis.map(toSQLId), + unresolvedFunc.origin) } } @@ -345,6 +367,25 @@ class FunctionResolution( } // Check external catalog for persistent functions + if (nameParts.length == 1) { + // Must match [[resolutionCandidates]] / [[resolveFunction]]: single-part names use PATH + + // session order, not only the current namespace (LookupCatalog single-part rule). + for (candidate <- resolutionCandidates(nameParts)) { + try { + candidate match { + case CatalogAndIdentifier(catalog, ident) => + if (catalog.asFunctionCatalog.functionExists(ident)) { + return FunctionType.Persistent + } + case _ => + } + } catch { + case NonFatal(_) => + } + } + return FunctionType.NotFound + } + val CatalogAndIdentifier(catalog, ident) = relationResolution.expandIdentifier(nameParts) if (catalog.asFunctionCatalog.functionExists(ident)) { return FunctionType.Persistent @@ -592,6 +633,66 @@ class FunctionResolution( errorClass = errorClass, messageParameters = messageParameters) } + + /** + * Resolves [[UnresolvedProcedure]] for `CALL` / `DESCRIBE PROCEDURE` using the same multipart + * candidates as SQL functions and relations ([[resolutionCandidates]] / + * [[sqlResolutionPathEntriesForAnalysis]]). Catalogs that do not implement + * [[ProcedureCatalog]] are skipped for unqualified names; an explicitly catalog-qualified name + * that targets a non-[[ProcedureCatalog]] still raises + * [[QueryCompilationErrors.missingCatalogProceduresAbilityError]]. + */ + def resolveProcedure(unresolved: UnresolvedProcedure): LogicalPlan = { + val candidates = resolutionCandidates(unresolved.nameParts) + val skipCandidateFailures = unresolved.nameParts.length == 1 + for (multipart <- candidates) { + val expandedOpt = + try { + Some(relationResolution.expandIdentifier(multipart)) + } catch { + case NonFatal(_) => None + } + expandedOpt.foreach { expanded => + CatalogAndIdentifier.unapply(expanded).foreach { case (catalog, ident) => + catalog match { + case pc: ProcedureCatalog => + try { + val procedure = pc.loadProcedure(ident) + return ResolvedProcedure(pc, ident, procedure) + } catch { + // ProcedureCatalog has no standard "not found" exception type today. For + // unqualified names searched through PATH, treat candidate failures as misses and + // continue to the next entry (matching table/function PATH iteration semantics). + // Explicitly catalog-qualified names still preserve existing error behavior. + case _: AnalysisException if skipCandidateFailures => + case _: SparkThrowable if skipCandidateFailures => + case NonFatal(_) if skipCandidateFailures => + case e: AnalysisException => throw e + case e: SparkThrowable => throw e + case NonFatal(e) => + val cause = e match { + case ex: Exception => ex + case th => new RuntimeException(th) + } + throw QueryCompilationErrors.failedToLoadRoutineError( + catalog.name +: ident.asMultipartIdentifier, + cause) + } + case _ => + if (unresolved.nameParts.length > 1 && + catalogManager.isCatalogRegistered(unresolved.nameParts.head) && + catalog.name().equalsIgnoreCase(unresolved.nameParts.head)) { + throw QueryCompilationErrors.missingCatalogProceduresAbilityError(catalog) + } + } + } + } + } + throw QueryCompilationErrors.unresolvedRoutineError( + unresolved.nameParts, + sqlResolutionPathEntriesForAnalysis.map(toSQLId), + unresolved.origin) + } } /** diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala index 58f832ea6cbdf..7a5077a8a3e11 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala @@ -116,34 +116,63 @@ class RelationResolution( /** * Scope in the relation resolution search path. Used to interpret - * [[SQLConf.resolutionSearchPath]] when resolving unqualified table/view names. + * [[CatalogManager.sqlResolutionPathEntries]] when resolving unqualified table/view names. */ - private sealed trait RelationResolutionScope - private case object SessionScope extends RelationResolutionScope - private case object PersistentScope extends RelationResolutionScope + private sealed trait RelationResolutionStep + private case object SessionScopeStep extends RelationResolutionStep + private case class PersistentCatalogStep(catalogAndNamespace: Seq[String]) + extends RelationResolutionStep /** - * Returns the relation resolution search path for unqualified (1-part) names. - * Uses the single search path for all objects: [[SQLConf.resolutionSearchPath]]. - * Maps path entries: system.session -> SessionScope, system.builtin -> skip (no views), - * other (catalog path) -> PersistentScope. + * Path entries for unqualified relation resolution. + * + * Inside a view, [[AnalysisContext.resolutionPathEntries]] will be + * populated from the frozen path stored in view metadata (follow-up PR). + * When PATH is disabled, legacy resolution rules apply. */ - private def relationResolutionSearchPath: Seq[RelationResolutionScope] = { - val catalogPath = (currentCatalog.name +: catalogManager.currentNamespace).toSeq - conf.resolutionSearchPath(catalogPath).flatMap { - case Seq("system", "session") => Some(SessionScope) + private def relationResolutionEntries: Seq[Seq[String]] = { + val pinned = AnalysisContext.get.resolutionPathEntries + if (pinned.isDefined && conf.pathEnabled) { + pinned.get + } else { + // Keep expanding CurrentSchemaEntry using the live session catalog/namespace until the + // follow-up PR wires frozen resolutionPathEntries for view analysis. + val expandCatalog = catalogManager.currentCatalog.name + val expandNamespace = catalogManager.currentNamespace.toSeq + val (pathCatalog, pathNamespace) = + if (isResolvingView) { + val p = AnalysisContext.get.catalogAndNamespace + (p.head, p.tail.toSeq) + } else { + (expandCatalog, expandNamespace) + } + catalogManager.sqlResolutionPathEntries( + pathCatalog, + pathNamespace, + expandCatalog, + expandNamespace) + } + } + + /** + * Ordered resolution steps for unqualified relation names. Each persistent path entry is kept + * with its catalog/namespace so lookup qualifies the object name under that entry (not only + * under the session's current namespace). + */ + private def relationResolutionSteps: Seq[RelationResolutionStep] = { + relationResolutionEntries.flatMap { + case p if CatalogManager.isSystemSessionPathEntry(p) => Some(SessionScopeStep) case Seq("system", "builtin") => None - case _ => Some(PersistentScope) + case entry => Some(PersistentCatalogStep(entry)) } } /** * Resolution search path formatted for TABLE_OR_VIEW_NOT_FOUND error messages. - * Same order as relationResolutionSearchPath; each entry is quoted (e.g. "`system`.`session`"). + * Same order as [[relationResolutionSteps]]; each entry is quoted (e.g. "`system`.`session`"). */ def resolutionSearchPathForError: Seq[String] = { - val catalogPath = (currentCatalog.name +: catalogManager.currentNamespace).toSeq - conf.resolutionSearchPath(catalogPath).map(toSQLId) + relationResolutionEntries.map(toSQLId) } /** @@ -201,15 +230,15 @@ class RelationResolution( ).orElse(tryResolvePersistent(u, identifier, finalTimeTravelSpec)) } - // 1-part name: try each scope in relationResolutionSearchPath order (from - // [[SQLConf.resolutionSearchPath]]). - val candidates = relationResolutionSearchPath - for (scope <- candidates) { - val result = scope match { - case SessionScope => + // 1-part name: try each step in [[relationResolutionSteps]] order (from + // [[CatalogManager.sqlResolutionPathEntries]]). + val steps = relationResolutionSteps + for (step <- steps) { + val result = step match { + case SessionScopeStep => resolveTempView(identifier, u.isStreaming, finalTimeTravelSpec.isDefined) - case PersistentScope => - tryResolvePersistent(u, identifier, finalTimeTravelSpec) + case PersistentCatalogStep(prefix) => + tryResolvePersistent(u, prefix ++ identifier, finalTimeTravelSpec) } if (result.isDefined) return result } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala index f34c6be9954e9..f7319e9b03e84 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala @@ -79,6 +79,12 @@ class ResolveCatalogs(val catalogManager: CatalogManager) throw new AnalysisException( "UNSUPPORTED_FEATURE.SQL_SCRIPTING_DROP_TEMPORARY_VARIABLE", Map.empty) } + if (nameParts.length == 1 && + !catalogManager.sessionScopeUnqualifiedAllowed( + catalogManager.currentCatalog.name(), + catalogManager.currentNamespace.toSeq)) { + throw QueryCompilationErrors.unresolvedVariableError(nameParts, Seq("SYSTEM", "SESSION")) + } val resolved = catalogManager.tempVariableManager.qualify(nameParts.last) assertValidSessionVariableNameParts(nameParts, resolved) d.copy(name = resolved) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala index 55622637f3046..b47332ace2b85 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala @@ -34,7 +34,8 @@ class ResolveFetchCursor(val catalogManager: CatalogManager) extends Rule[Logica with ColumnResolutionHelper { // VariableResolution looks up both scripting local variables (via SqlScriptingContextManager) // and session variables (via tempVariableManager), checking local variables first. - private val variableResolution = new VariableResolution(catalogManager.tempVariableManager) + private val variableResolution = + new VariableResolution(catalogManager.tempVariableManager, catalogManager) /** * Checks for duplicate variable names and throws an exception if found. diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSetVariable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSetVariable.scala index 4b16448641bc1..6ecbc87d35530 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSetVariable.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSetVariable.scala @@ -33,7 +33,8 @@ import org.apache.spark.sql.types.IntegerType */ class ResolveSetVariable(val catalogManager: CatalogManager) extends Rule[LogicalPlan] with ColumnResolutionHelper { - private val variableResolution = new VariableResolution(catalogManager.tempVariableManager) + private val variableResolution = + new VariableResolution(catalogManager.tempVariableManager, catalogManager) /** * Checks for duplicate variable names and throws an exception if found. diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/VariableResolution.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/VariableResolution.scala index 0095885c0135d..f8cce0d6f821e 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/VariableResolution.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/VariableResolution.scala @@ -33,7 +33,21 @@ import org.apache.spark.sql.connector.catalog.{ Identifier } -class VariableResolution(tempVariableManager: TempVariableManager) extends SQLConfHelper { +class VariableResolution( + tempVariableManager: TempVariableManager, + catalogManager: CatalogManager) + extends SQLConfHelper { + + /** + * Unqualified session variables resolve only when SYSTEM.SESSION is on the SQL path + * (PATH enabled and explicitly set). + */ + private def allowUnqualifiedSessionTempVariableLookup(nameParts: Seq[String]): Boolean = { + if (nameParts.length != 1) return true + catalogManager.sessionScopeUnqualifiedAllowed( + catalogManager.currentCatalog.name(), + catalogManager.currentNamespace.toSeq) + } /** * Resolves a `multipartName` to an [[Expression]] tree, supporting nested field access. @@ -125,7 +139,8 @@ class VariableResolution(tempVariableManager: TempVariableManager) extends SQLCo ) } .orElse( - if (maybeTempVariableName(nameParts)) { + if (maybeTempVariableName(nameParts) && + allowUnqualifiedSessionTempVariableLookup(nameParts)) { tempVariableManager .get(namePartsCaseAdjusted) .map { varDef => diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/NameScope.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/NameScope.scala index 1c9a296af9ba4..2c39fe71e62ec 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/NameScope.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/NameScope.scala @@ -44,6 +44,7 @@ import org.apache.spark.sql.catalyst.expressions.{ } import org.apache.spark.sql.catalyst.plans.logical.Aggregate import org.apache.spark.sql.catalyst.util._ +import org.apache.spark.sql.connector.catalog.CatalogManager import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.Metadata @@ -974,13 +975,14 @@ class NameScope( */ class NameScopeStack( tempVariableManager: TempVariableManager, + catalogManager: CatalogManager, subqueryRegistry: SubqueryRegistry, planLogger: PlanLogger = new PlanLogger) extends SQLConfHelper { private val stack = new ArrayDeque[NameScope] stack.push(new NameScope(planLogger = planLogger)) - private val variableResolution = new VariableResolution(tempVariableManager) + private val variableResolution = new VariableResolution(tempVariableManager, catalogManager) /** * Get the current scope, which is a default choice for name resolution. diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala index 35d752779d6c6..aaf7117ef4e8b 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala @@ -26,6 +26,7 @@ import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.QueryPlanningTracker import org.apache.spark.sql.catalyst.analysis.{ withPosition, + AnalysisContext, AnalysisErrorAt, CleanupAliases, FunctionResolution, @@ -53,8 +54,7 @@ import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.catalyst.trees.CurrentOrigin import org.apache.spark.sql.catalyst.util.EvaluateUnresolvedInlineTable import org.apache.spark.sql.connector.catalog.CatalogManager -import org.apache.spark.sql.errors.QueryCompilationErrors -import org.apache.spark.sql.errors.QueryErrorsBase +import org.apache.spark.sql.errors.{QueryCompilationErrors, QueryErrorsBase} import org.apache.spark.sql.internal.SQLConf /** @@ -96,6 +96,7 @@ class Resolver( private val subqueryRegistry = new SubqueryRegistry private val scopes = new NameScopeStack( tempVariableManager = catalogManager.tempVariableManager, + catalogManager = catalogManager, subqueryRegistry = subqueryRegistry, planLogger = planLogger ) @@ -583,9 +584,14 @@ class Resolver( relationsWithResolvedMetadata case None => val multipartId = unresolvedRelation.multipartIdentifier - val catalogPath = (catalogManager.currentCatalog.name() +: - catalogManager.currentNamespace).toSeq - val searchPath = SQLConf.get.resolutionSearchPath(catalogPath).map(toSQLId) + val catalogPath = { + val ctx = AnalysisContext.get.catalogAndNamespace + if (ctx.nonEmpty) ctx + else (catalogManager.currentCatalog.name() +: catalogManager.currentNamespace).toSeq + } + val searchPath = catalogManager + .sqlResolutionPathEntries(catalogPath.head, catalogPath.tail.toSeq) + .map(toSQLId) unresolvedRelation.tableNotFound(multipartId, searchPath) } @@ -842,7 +848,7 @@ class Resolver( messageParameters = Map( "missingAttributes" -> makeCommaSeparatedExpressionString(missingInput.toSeq), "input" -> makeCommaSeparatedExpressionString(inputSet.toSeq), - "operator" -> operator.simpleString(conf.maxToStringFields), + "operator" -> operator.simpleString(SQLConf.get.maxToStringFields), "operation" -> makeCommaSeparatedExpressionString(attributesWithSameName.toSeq) ) ) @@ -852,7 +858,7 @@ class Resolver( messageParameters = Map( "missingAttributes" -> makeCommaSeparatedExpressionString(missingInput.toSeq), "input" -> makeCommaSeparatedExpressionString(inputSet.toSeq), - "operator" -> operator.simpleString(conf.maxToStringFields) + "operator" -> operator.simpleString(SQLConf.get.maxToStringFields) ) ) } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala index e2171b84b6eb2..ed41c320f8c9b 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala @@ -474,9 +474,8 @@ class ResolverGuard( private def checkUnresolvedFunction(unresolvedFunction: UnresolvedFunction) = { val nameParts = unresolvedFunction.nameParts val funcName = nameParts.last.toLowerCase(Locale.ROOT) - - if (nameParts.length == 1) { - // Unqualified: same as master (unsupported, non-builtin, or check children) + if (nameParts.size == 1) { + // Unqualified: reject if unsupported, else non-builtin or check children (same as master) if (isUnsupportedFunction(funcName)) { Some(s"unsupported function ${funcName}") } else if (!isBuiltinFunction(funcName)) { @@ -493,7 +492,7 @@ class ResolverGuard( unresolvedFunction.children.collectFirst { case CheckExpression(reason) => reason } } } else if (FunctionResolution.sessionNamespaceKind(nameParts).isDefined) { - // Session-qualified: allow through (system-first behavior) + // Session-qualified: allow through (PATH + system-first) unresolvedFunction.children.collectFirst { case CheckExpression(reason) => reason } } else { Some("multi-part function name") diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/v2ResolutionPlans.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/v2ResolutionPlans.scala index fd05b7cd5a2a7..046acd20a9e3d 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/v2ResolutionPlans.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/v2ResolutionPlans.scala @@ -68,15 +68,37 @@ case class UnresolvedView( allowTemp: Boolean, suggestAlternative: Boolean = false) extends UnresolvedLeafNode +/** + * Controls which search path is shown in `TABLE_OR_VIEW_NOT_FOUND` for + * [[UnresolvedTableOrView]] (see [[org.apache.spark.sql.catalyst.analysis.CheckAnalysis]]). + */ +sealed trait UnresolvedTableOrViewSearchPathMode + +object UnresolvedTableOrViewSearchPathMode { + /** DDL on catalog objects: `system.session` and current catalog namespace only. */ + case object Ddl extends UnresolvedTableOrViewSearchPathMode + /** + * Like `SELECT` / DML: full `sqlResolutionPathEntries` order; fully qualified + * `system.session.*` names still use the session-only path in errors. + */ + case object QueryLike extends UnresolvedTableOrViewSearchPathMode +} + /** * Holds the name of a table or view that has yet to be looked up in a catalog. It will * be resolved to [[ResolvedTable]], [[ResolvedPersistentView]] or [[ResolvedTempView]] during * analysis. + * + * @param tableNotFoundSearchPathMode how to format `searchPath` in `TABLE_OR_VIEW_NOT_FOUND`; + * set explicitly at parse / construction time (not inferred + * from [[commandName]]). */ case class UnresolvedTableOrView( multipartIdentifier: Seq[String], commandName: String, - allowTempView: Boolean) extends UnresolvedLeafNode + allowTempView: Boolean, + tableNotFoundSearchPathMode: UnresolvedTableOrViewSearchPathMode = + UnresolvedTableOrViewSearchPathMode.Ddl) extends UnresolvedLeafNode sealed trait PartitionSpec extends LeafExpression with Unevaluable { override def dataType: DataType = throw SparkException.internalError( diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala index 527f2b80314df..df83f558c892e 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala @@ -4286,8 +4286,12 @@ class AstBuilder extends DataTypeAstBuilder protected def createUnresolvedTableOrView( ctx: IdentifierReferenceContext, commandName: String, - allowTempView: Boolean = true): LogicalPlan = withOrigin(ctx) { - withIdentClause(ctx, UnresolvedTableOrView(_, commandName, allowTempView)) + allowTempView: Boolean = true, + tableNotFoundSearchPathMode: UnresolvedTableOrViewSearchPathMode = + UnresolvedTableOrViewSearchPathMode.Ddl): LogicalPlan = withOrigin(ctx) { + withIdentClause( + ctx, + UnresolvedTableOrView(_, commandName, allowTempView, tableNotFoundSearchPathMode)) } private def createUnresolvedTableOrView( diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala index 7df836cea6124..3f5afd9ce0de7 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala @@ -164,6 +164,43 @@ class CatalogManager( } } + /** + * Ordered catalog/schema path entries for resolving unqualified SQL object names. + * When PATH is off or unset, applies [[SQLConf.defaultPathOrder]] (legacy). + * When PATH is explicitly set, uses the resolved stored path entries. + */ + def sqlResolutionPathEntries( + pathDefaultCatalog: String, + pathDefaultNamespace: Seq[String], + expandCatalog: String, + expandNamespace: Seq[String]): Seq[Seq[String]] = synchronized { + val defaultEntry = + if (pathDefaultNamespace.isEmpty) Seq(pathDefaultCatalog) + else pathDefaultCatalog +: pathDefaultNamespace + val stored = if (conf.pathEnabled) _sessionPath else None + stored match { + case Some(entries) => + CatalogManager.resolvePathEntries(entries, expandCatalog, expandNamespace) + case None => + conf.defaultPathOrder(Seq(defaultEntry)) + } + } + + /** Session-catalog overload. */ + def sqlResolutionPathEntries( + currentCatalog: String, + currentNamespace: Seq[String]): Seq[Seq[String]] = + sqlResolutionPathEntries( + currentCatalog, currentNamespace, + currentCatalog, currentNamespace) + + /** True if [[sqlResolutionPathEntries]] includes `system.session`. */ + def sessionScopeUnqualifiedAllowed( + currentCatalog: String, + currentNamespace: Seq[String]): Boolean = + sqlResolutionPathEntries(currentCatalog, currentNamespace) + .exists(CatalogManager.isSystemSessionPathEntry) + private var _currentCatalogName: Option[String] = None def currentCatalog: CatalogPlugin = synchronized { diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TableLookupCacheSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TableLookupCacheSuite.scala index 9685ed5c6d256..63d5523be072d 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TableLookupCacheSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TableLookupCacheSuite.scala @@ -33,6 +33,7 @@ import org.apache.spark.sql.catalyst.dsl.plans._ import org.apache.spark.sql.connector.catalog.{CatalogManager, Identifier, InMemoryTable, InMemoryTableCatalog, Table} import org.apache.spark.sql.connector.catalog.TableWritePrivilege import org.apache.spark.sql.errors.QueryExecutionErrors +import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types._ class TableLookupCacheSuite extends AnalysisTest with Matchers { @@ -74,6 +75,14 @@ class TableLookupCacheSuite extends AnalysisTest with Matchers { when(catalogManager.v1SessionCatalog).thenReturn(v1Catalog) when(catalogManager.currentCatalog).thenReturn(v2Catalog) when(catalogManager.currentNamespace).thenReturn(Array("default")) + when(catalogManager.sessionPathEntries).thenReturn(None) + val defaultPath = SQLConf.get.resolutionSearchPath( + (v2Catalog.name() +: Array("default")).toSeq) + when(catalogManager.sqlResolutionPathEntries( + any[String], any[Seq[String]], any[String], any[Seq[String]])) + .thenReturn(defaultPath) + when(catalogManager.sqlResolutionPathEntries(any[String], any[Seq[String]])) + .thenReturn(defaultPath) new Analyzer(catalogManager) } diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolverSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolverSuite.scala index 8897d65654540..f54ab9e4e0ddd 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolverSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolverSuite.scala @@ -37,7 +37,9 @@ class TimezoneAwareExpressionResolverSuite extends SparkFunSuite { extends ExpressionResolver( resolver = new Resolver(catalogManager), functionResolution = - new FunctionResolution(catalogManager, Resolver.createRelationResolution(catalogManager)), + new FunctionResolution( + catalogManager, + Resolver.createRelationResolution(catalogManager)), planLogger = new PlanLogger ) { override def resolve(expression: Expression): Expression = resolvedExpression diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/read_changes.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/read_changes.explain index 413eaf8f7a686..9de2e887e1ee6 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/read_changes.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/read_changes.explain @@ -1,2 +1,2 @@ -SubqueryAlias primary.tempdb.myTable -+- RelationV2[id#0L, _change_type#0, _commit_version#0L, _commit_timestamp#0] primary.tempdb.myTable +SubqueryAlias spark_catalog.tempdb.myTable ++- RelationV2[id#0L, _change_type#0, _commit_version#0L, _commit_timestamp#0] spark_catalog.tempdb.myTable diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/read_changes_with_options.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/read_changes_with_options.explain index 413eaf8f7a686..9de2e887e1ee6 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/read_changes_with_options.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/read_changes_with_options.explain @@ -1,2 +1,2 @@ -SubqueryAlias primary.tempdb.myTable -+- RelationV2[id#0L, _change_type#0, _commit_version#0L, _commit_timestamp#0] primary.tempdb.myTable +SubqueryAlias spark_catalog.tempdb.myTable ++- RelationV2[id#0L, _change_type#0, _commit_version#0L, _commit_timestamp#0] spark_catalog.tempdb.myTable diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/read_table.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/read_table.explain index 979084f06a87a..e5dce0fe05742 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/read_table.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/read_table.explain @@ -1,2 +1,2 @@ -SubqueryAlias primary.tempdb.myTable -+- RelationV2[id#0L] primary.tempdb.myTable +SubqueryAlias spark_catalog.tempdb.myTable ++- RelationV2[id#0L] spark_catalog.tempdb.myTable diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_changes_API_with_options.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_changes_API_with_options.explain index 6f12567607ac0..a0316c9dba3a3 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_changes_API_with_options.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_changes_API_with_options.explain @@ -1,2 +1,2 @@ -~SubqueryAlias primary.tempdb.myStreamingTable -+- ~StreamingRelationV2 primary.tempdb.myStreamingTable_changelog, ChangelogTable(org.apache.spark.sql.connector.catalog.InMemoryChangelog,ChangelogInfo{range=VersionRange[startingVersion=1, endingVersion=Optional.empty, startingBoundInclusive=true, endingBoundInclusive=true], deduplicationMode=DROP_CARRYOVERS, computeUpdates=false}), [startingVersion=1, deduplicationMode=dropCarryovers], [id#0L, _change_type#0, _commit_version#0L, _commit_timestamp#0], org.apache.spark.sql.connector.catalog.InMemoryChangelogCatalog, tempdb.myStreamingTable, name= +~SubqueryAlias spark_catalog.tempdb.myStreamingTable ++- ~StreamingRelationV2 spark_catalog.tempdb.myStreamingTable_changelog, ChangelogTable(org.apache.spark.sql.connector.catalog.InMemoryChangelog,ChangelogInfo{range=VersionRange[startingVersion=1, endingVersion=Optional.empty, startingBoundInclusive=true, endingBoundInclusive=true], deduplicationMode=DROP_CARRYOVERS, computeUpdates=false}), [startingVersion=1, deduplicationMode=dropCarryovers], [id#0L, _change_type#0, _commit_version#0L, _commit_timestamp#0], org.apache.spark.sql.connector.catalog.InMemoryChangelogCatalog, tempdb.myStreamingTable, name= diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_table_API_with_options.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_table_API_with_options.explain index 9ea4ad218a5f4..dc17d3503894d 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_table_API_with_options.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_table_API_with_options.explain @@ -1,2 +1,2 @@ -~SubqueryAlias primary.tempdb.myStreamingTable -+- ~StreamingRelationV2 primary.tempdb.myStreamingTable, org.apache.spark.sql.connector.catalog.InMemoryTable, [p1=v1, p2=v2], [id#0L], org.apache.spark.sql.connector.catalog.InMemoryChangelogCatalog, tempdb.myStreamingTable, name= +~SubqueryAlias spark_catalog.tempdb.myStreamingTable ++- ~StreamingRelationV2 spark_catalog.tempdb.myStreamingTable, org.apache.spark.sql.connector.catalog.InMemoryTable, [p1=v1, p2=v2], [id#0L], org.apache.spark.sql.connector.catalog.InMemoryChangelogCatalog, tempdb.myStreamingTable, name= diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/table.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/table.explain index 979084f06a87a..e5dce0fe05742 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/table.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/table.explain @@ -1,2 +1,2 @@ -SubqueryAlias primary.tempdb.myTable -+- RelationV2[id#0L] primary.tempdb.myTable +SubqueryAlias spark_catalog.tempdb.myTable ++- RelationV2[id#0L] spark_catalog.tempdb.myTable diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/table_API_with_options.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/table_API_with_options.explain index 979084f06a87a..e5dce0fe05742 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/table_API_with_options.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/table_API_with_options.explain @@ -1,2 +1,2 @@ -SubqueryAlias primary.tempdb.myTable -+- RelationV2[id#0L] primary.tempdb.myTable +SubqueryAlias spark_catalog.tempdb.myTable ++- RelationV2[id#0L] spark_catalog.tempdb.myTable diff --git a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala index 506a75ca3b4c3..fabbf3071c4ec 100644 --- a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala +++ b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala @@ -28,7 +28,7 @@ import org.apache.spark.{SparkConf, SparkFunSuite} import org.apache.spark.connect.proto import org.apache.spark.internal.LogKeys.PATH import org.apache.spark.sql.catalyst.{catalog, QueryPlanningTracker} -import org.apache.spark.sql.catalyst.analysis.{caseSensitiveResolution, Analyzer, FunctionRegistry, Resolver, TableFunctionRegistry} +import org.apache.spark.sql.catalyst.analysis.{caseSensitiveResolution, Analyzer, FunctionRegistry, RelationCache, Resolver, TableFunctionRegistry} import org.apache.spark.sql.catalyst.catalog.SessionCatalog import org.apache.spark.sql.catalyst.optimizer.{ReplaceExpressions, RewriteWithExpression} import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan @@ -133,9 +133,25 @@ class ProtoToParsedPlanTestSuite protected val goldenFilePath: Path = suiteBaseResourcePath.resolve("explain-results") private val emptyProps: util.Map[String, String] = util.Collections.emptyMap() + /** + * Isolated from [[SharedSparkSession]] so PATH / session path settings do not affect catalog. + */ + private val analyzerIsolationConf: SQLConf = { + val c = new SQLConf() + c.setConf(SQLConf.PATH_ENABLED, false) + // Match [[sparkConf]]: a bare SQLConf defaults ANSI_ENABLED to true, which changes + // function signatures in analyzed plans (e.g. make_date) vs golden files. + c.setConf(SQLConf.ANSI_ENABLED, false) + c + } + private val analyzer = { val inMemoryCatalog = new InMemoryChangelogCatalog - inMemoryCatalog.initialize("primary", CaseInsensitiveStringMap.empty()) + // Name must match [[CatalogManager.SESSION_CATALOG_NAME]]: path entries use + // [[currentCatalog.name()]], then resolution calls [[catalogManager.catalog]] on that segment. + inMemoryCatalog.initialize( + CatalogManager.SESSION_CATALOG_NAME, + CaseInsensitiveStringMap.empty()) inMemoryCatalog.createNamespace(Array("tempdb"), emptyProps) inMemoryCatalog.createTable( Identifier.of(Array("tempdb"), "myTable"), @@ -154,10 +170,12 @@ class ProtoToParsedPlanTestSuite new catalog.InMemoryCatalog(), FunctionRegistry.builtin, TableFunctionRegistry.builtin)) - catalogManager.setCurrentCatalog("primary") + // Do not call setCurrentCatalog("primary"): that loads a separate plugin via + // Catalogs.load("primary", conf) instead of using defaultSessionCatalog (inMemoryCatalog). + // Leave current catalog as default spark_catalog so v2SessionCatalog returns inMemoryCatalog. catalogManager.setCurrentNamespace(Array("tempdb")) - new Analyzer(catalogManager) { + new Analyzer(catalogManager, RelationCache.empty, Some(analyzerIsolationConf)) { override def resolver: Resolver = caseSensitiveResolution } } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/classic/Catalog.scala b/sql/core/src/main/scala/org/apache/spark/sql/classic/Catalog.scala index 35041feca9e18..9a5aed333a4c2 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/classic/Catalog.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/classic/Catalog.scala @@ -409,10 +409,12 @@ class Catalog(sparkSession: SparkSession) extends catalog.Catalog with Logging { val catalogPath = (Seq(currentCatalog()) ++ sparkSession.sessionState.catalogManager.currentNamespace).toSeq - val searchPath = sparkSession.sessionState.conf.resolutionSearchPath(catalogPath) + val searchPath = sparkSession.sessionState.catalogManager + .sqlResolutionPathEntries(catalogPath.head, catalogPath.tail.toSeq) + .map(_.quoted) throw QueryCompilationErrors.unresolvedRoutineError( ident, - searchPath.map(_.quoted), + searchPath, plan.origin) } } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala index b910a3dd6d8aa..8f4c77840f0cc 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala @@ -30,7 +30,8 @@ import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier} import org.apache.spark.sql.catalyst.analysis.{CurrentNamespace, GlobalTempView, LocalTempView, PersistedView, PlanWithUnresolvedIdentifier, SchemaEvolution, SchemaTypeEvolution, UnresolvedAttribute, - UnresolvedIdentifier, UnresolvedNamespace, UnresolvedPartitionSpec, UnresolvedProcedure} + UnresolvedIdentifier, UnresolvedNamespace, UnresolvedPartitionSpec, UnresolvedProcedure, + UnresolvedTableOrViewSearchPathMode} import org.apache.spark.sql.catalyst.catalog._ import org.apache.spark.sql.catalyst.expressions.{Expression, Literal} import org.apache.spark.sql.catalyst.parser._ @@ -1450,7 +1451,11 @@ class SparkSqlAstBuilder extends AstBuilder { val tableName = ctx.identifierReference.getText.split("\\.").lastOption.getOrElse("table") throw QueryCompilationErrors.describeJsonNotExtendedError(tableName) } - val relation = createUnresolvedTableOrView(ctx.identifierReference, "DESCRIBE TABLE") + val relation = createUnresolvedTableOrView( + ctx.identifierReference, + "DESCRIBE TABLE", + allowTempView = true, + UnresolvedTableOrViewSearchPathMode.QueryLike) if (ctx.describeColName != null) { if (ctx.partitionSpec != null) { throw QueryParsingErrors.descColumnForPartitionUnsupportedError(ctx) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala index e248f0eea96de..4a9bebe75cff1 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala @@ -108,8 +108,8 @@ case class SetCommand(kv: Option[(String, Option[String])]) } if (varName.nonEmpty && varName.length <= 3) { val variableResolution = new VariableResolution( - sparkSession.sessionState.analyzer.catalogManager.tempVariableManager - ) + sparkSession.sessionState.analyzer.catalogManager.tempVariableManager, + sparkSession.sessionState.analyzer.catalogManager) val variable = variableResolution.lookupVariable( nameParts = varName ) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala b/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala index d8fe14a0664c1..cc8c9dcb71f5d 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala @@ -189,7 +189,7 @@ abstract class BaseSessionStateBuilder( * * Note: this depends on the `conf` and `catalog` fields. */ - protected def analyzer: Analyzer = new Analyzer(catalogManager, sharedRelationCache) { + protected def analyzer: Analyzer = new Analyzer(catalogManager, sharedRelationCache, Some(conf)) { override val hintResolutionRules: Seq[Rule[LogicalPlan]] = customHintResolutionRules diff --git a/sql/core/src/test/resources/sql-tests/results/describe.sql.out b/sql/core/src/test/resources/sql-tests/results/describe.sql.out index 36985a0ec628a..3d5db15a48113 100644 --- a/sql/core/src/test/resources/sql-tests/results/describe.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/describe.sql.out @@ -707,7 +707,7 @@ struct -- !query output == Parsed Logical Plan == 'DescribeRelation false, [col_name#x, data_type#x, comment#x] -+- 'UnresolvedTableOrView [t], DESCRIBE TABLE, true ++- 'UnresolvedTableOrView [t], DESCRIBE TABLE, true, QueryLike == Analyzed Logical Plan == col_name: string, data_type: string, comment: string diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala index eb3815f6209c8..c24046e818e91 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala @@ -320,18 +320,22 @@ class SetPathSuite extends SharedSparkSession { test("PATH enabled: multi-level namespace (3+ parts) is accepted") { withPathEnabled { - sql("SET PATH = iceberg_cat.db1.db2, spark_catalog.default") - val entries = pathEntries(currentPath()) - assert(entries.head === "iceberg_cat.db1.db2", - s"Multi-level namespace should be accepted; got: $entries") + // SET PATH should accept multi-level namespaces without error. + // We verify the path is stored correctly via the CatalogManager API + // rather than currentPath(), which would fail because spark_catalog + // only supports single-part namespaces. + sql("SET PATH = spark_catalog.ns1.ns2, spark_catalog.default") + val stored = spark.sessionState.catalogManager.sessionPathEntries + assert(stored.isDefined, "Session path should be stored") + assert(stored.get.length == 2, s"Should have 2 entries, got: ${stored.get}") } } test("PATH enabled: backtick-quoted identifiers with dots round-trip correctly") { withPathEnabled { - sql("SET PATH = `cat.a`.`sch.b`") + sql("SET PATH = spark_catalog.`sch.b`, system.builtin") val entries = pathEntries(currentPath()) - assert(entries === Seq("`cat.a`.`sch.b`"), + assert(entries.head === "spark_catalog.`sch.b`", s"Backtick-quoted identifiers should round-trip; got: $entries") } } @@ -382,6 +386,68 @@ class SetPathSuite extends SharedSparkSession { // TODO: cloneSession() constructs a new CatalogManager per forked session and // explicitly copies only the stored session path via copySessionPathFrom. // Other CatalogManager state propagation (current catalog/namespace, registered - // catalogs) on clone is currently incidental — audit and pin down the intended + // catalogs) on clone is currently incidental -- audit and pin down the intended // semantics in a follow-up. + + // --- Resolution tests: verify SET PATH affects actual table/function lookup --- + + test("PATH enabled: table resolves from first matching path entry") { + withPathEnabled { + sql("CREATE SCHEMA IF NOT EXISTS path_res_a") + sql("CREATE SCHEMA IF NOT EXISTS path_res_b") + sql("CREATE TABLE path_res_a.tbl (x INT) USING parquet") + sql("CREATE TABLE path_res_b.tbl (x INT) USING parquet") + sql("INSERT INTO path_res_a.tbl VALUES (1)") + sql("INSERT INTO path_res_b.tbl VALUES (2)") + try { + sql("SET PATH = spark_catalog.path_res_a, spark_catalog.path_res_b, system.builtin") + checkAnswer(sql("SELECT x FROM tbl"), Row(1)) + sql("SET PATH = spark_catalog.path_res_b, spark_catalog.path_res_a, system.builtin") + checkAnswer(sql("SELECT x FROM tbl"), Row(2)) + } finally { + sql("DROP TABLE IF EXISTS path_res_a.tbl") + sql("DROP TABLE IF EXISTS path_res_b.tbl") + sql("DROP SCHEMA IF EXISTS path_res_a") + sql("DROP SCHEMA IF EXISTS path_res_b") + } + } + } + + test("PATH enabled: function resolves from first matching path entry") { + withPathEnabled { + sql("CREATE SCHEMA IF NOT EXISTS path_fn_a") + sql("CREATE SCHEMA IF NOT EXISTS path_fn_b") + sql("CREATE FUNCTION path_fn_a.pick() RETURNS INT RETURN 1") + sql("CREATE FUNCTION path_fn_b.pick() RETURNS INT RETURN 2") + try { + sql("SET PATH = spark_catalog.path_fn_a, spark_catalog.path_fn_b, system.builtin") + checkAnswer(sql("SELECT pick()"), Row(1)) + sql("SET PATH = spark_catalog.path_fn_b, spark_catalog.path_fn_a, system.builtin") + checkAnswer(sql("SELECT pick()"), Row(2)) + } finally { + sql("DROP FUNCTION IF EXISTS path_fn_a.pick") + sql("DROP FUNCTION IF EXISTS path_fn_b.pick") + sql("DROP SCHEMA IF EXISTS path_fn_a") + sql("DROP SCHEMA IF EXISTS path_fn_b") + } + } + } + + test("PATH enabled: unqualified table fails when schema not in path") { + withPathEnabled { + sql("CREATE SCHEMA IF NOT EXISTS path_miss") + sql("CREATE TABLE path_miss.hidden (x INT) USING parquet") + try { + sql("SET PATH = spark_catalog.default, system.builtin") + val err = intercept[AnalysisException] { + sql("SELECT * FROM hidden") + } + assert(err.getMessage.contains("TABLE_OR_VIEW_NOT_FOUND"), + s"Expected TABLE_OR_VIEW_NOT_FOUND, got: ${err.getMessage}") + } finally { + sql("DROP TABLE IF EXISTS path_miss.hidden") + sql("DROP SCHEMA IF EXISTS path_miss") + } + } + } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/NameScopeSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/NameScopeSuite.scala index 30f587f6480d9..458e8be1dfe0e 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/NameScopeSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/NameScopeSuite.scala @@ -1219,6 +1219,7 @@ class NameScopeSuite extends SharedSparkSession { private def newNameScopeStack() = new NameScopeStack( tempVariableManager = spark.sessionState.analyzer.catalogManager.tempVariableManager, + catalogManager = spark.sessionState.analyzer.catalogManager, subqueryRegistry = new SubqueryRegistry ) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ProcedureSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ProcedureSuite.scala index 11bce7afb6545..f6b0dae9b362f 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ProcedureSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ProcedureSuite.scala @@ -222,6 +222,18 @@ class ProcedureSuite extends SharedSparkSession with BeforeAndAfter { } } + test("PATH enabled: unqualified CALL skips missing candidate and keeps searching") { + withSQLConf(SQLConf.PATH_ENABLED.key -> "true") { + try { + catalog("cat2").createProcedure(Identifier.of(Array("ns_hit"), "sum"), UnboundLongSum) + sql("SET PATH = cat.ns_miss, cat2.ns_hit") + checkAnswer(sql("CALL sum(1, 2)"), Row(3L) :: Nil) + } finally { + sql("SET PATH = DEFAULT_PATH") + } + } + } + test("required parameter not found") { catalog.createProcedure(Identifier.of(Array("ns"), "sum"), UnboundSum) checkError( diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlignAssignmentsSuiteBase.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlignAssignmentsSuiteBase.scala index 14cf72c78dbee..1a6dc178b6e51 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlignAssignmentsSuiteBase.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlignAssignmentsSuiteBase.scala @@ -188,6 +188,13 @@ abstract class AlignAssignmentsSuiteBase extends AnalysisTest { when(manager.v1SessionCatalog).thenReturn(v1SessionCatalog) when(manager.v2SessionCatalog).thenReturn(v2SessionCatalog) when(manager.tempVariableManager).thenReturn(tempVariableManager) + when(manager.sessionPathEntries).thenReturn(None) + val defaultPath = SQLConf.get.resolutionSearchPath(Seq(v2Catalog.name())) + when(manager.sqlResolutionPathEntries( + any[String], any[Seq[String]], any[String], any[Seq[String]])) + .thenReturn(defaultPath) + when(manager.sqlResolutionPathEntries(any[String], any[Seq[String]])) + .thenReturn(defaultPath) manager } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala index d9b91946ded77..9997b6c7bb385 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala @@ -731,7 +731,7 @@ class DDLParserSuite extends AnalysisTest with SharedSparkSession { parser.parsePlan(sql).collect { case CreateTableLike( UnresolvedIdentifier(targetParts, _), - UnresolvedTableOrView(sourceParts, _, _), + UnresolvedTableOrView(sourceParts, _, _, _), loc, p, _, pr, e) => (targetParts, sourceParts, loc, p, pr, e) }.head diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DescribeTableParserSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DescribeTableParserSuite.scala index 436fa2e2389aa..ebe8eaf91d56a 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DescribeTableParserSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DescribeTableParserSuite.scala @@ -19,33 +19,40 @@ package org.apache.spark.sql.execution.command import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedAttribute, - UnresolvedPartitionSpec, UnresolvedTableOrView} + UnresolvedPartitionSpec, UnresolvedTableOrView, UnresolvedTableOrViewSearchPathMode} import org.apache.spark.sql.catalyst.plans.logical.{DescribeColumn, DescribeRelation, DescribeTablePartition} import org.apache.spark.sql.test.SharedSparkSession class DescribeTableParserSuite extends SharedSparkSession with AnalysisTest { private def parsePlan(statement: String) = spark.sessionState.sqlParser.parsePlan(statement) + private def unresolvedDescribeTable(name: String): UnresolvedTableOrView = { + UnresolvedTableOrView( + Seq(name), + "DESCRIBE TABLE", + allowTempView = true, + UnresolvedTableOrViewSearchPathMode.QueryLike) + } test("SPARK-17328: Fix NPE with EXPLAIN DESCRIBE TABLE") { comparePlans(parsePlan("describe t"), DescribeRelation( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), isExtended = false)) + unresolvedDescribeTable("t"), isExtended = false)) comparePlans(parsePlan("describe table t"), DescribeRelation( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), isExtended = false)) + unresolvedDescribeTable("t"), isExtended = false)) comparePlans(parsePlan("describe table extended t"), DescribeRelation( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), isExtended = true)) + unresolvedDescribeTable("t"), isExtended = true)) comparePlans(parsePlan("describe table formatted t"), DescribeRelation( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), isExtended = true)) + unresolvedDescribeTable("t"), isExtended = true)) } test("describe table with partition spec") { comparePlans(parsePlan("DESCRIBE TABLE t PARTITION (ds='2024-01-01')"), DescribeTablePartition( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), + unresolvedDescribeTable("t"), UnresolvedPartitionSpec(Map("ds" -> "2024-01-01")), isExtended = false)) } @@ -53,38 +60,38 @@ class DescribeTableParserSuite extends SharedSparkSession with AnalysisTest { test("describe table column") { comparePlans(parsePlan("DESCRIBE t col"), DescribeColumn( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), + unresolvedDescribeTable("t"), UnresolvedAttribute(Seq("col")), isExtended = false)) comparePlans(parsePlan("DESCRIBE t `abc.xyz`"), DescribeColumn( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), + unresolvedDescribeTable("t"), UnresolvedAttribute(Seq("abc.xyz")), isExtended = false)) comparePlans(parsePlan("DESCRIBE t abc.xyz"), DescribeColumn( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), + unresolvedDescribeTable("t"), UnresolvedAttribute(Seq("abc", "xyz")), isExtended = false)) comparePlans(parsePlan("DESCRIBE t `a.b`.`x.y`"), DescribeColumn( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), + unresolvedDescribeTable("t"), UnresolvedAttribute(Seq("a.b", "x.y")), isExtended = false)) comparePlans(parsePlan("DESCRIBE TABLE t col"), DescribeColumn( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), + unresolvedDescribeTable("t"), UnresolvedAttribute(Seq("col")), isExtended = false)) comparePlans(parsePlan("DESCRIBE TABLE EXTENDED t col"), DescribeColumn( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), + unresolvedDescribeTable("t"), UnresolvedAttribute(Seq("col")), isExtended = true)) comparePlans(parsePlan("DESCRIBE TABLE FORMATTED t col"), DescribeColumn( - UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), + unresolvedDescribeTable("t"), UnresolvedAttribute(Seq("col")), isExtended = true)) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala index b564cad0fe9c8..a4b2ef49f0a26 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala @@ -221,6 +221,13 @@ class PlanResolutionSuite extends SharedSparkSession with AnalysisTest { when(manager.currentNamespace).thenReturn(Array.empty[String]) when(manager.v1SessionCatalog).thenReturn(v1SessionCatalog) when(manager.tempVariableManager).thenReturn(tempVariableManager) + when(manager.sessionPathEntries).thenReturn(None) + val defaultPath = SQLConf.get.resolutionSearchPath(Seq(testCat.name())) + when(manager.sqlResolutionPathEntries( + any[String], any[Seq[String]], any[String], any[Seq[String]])) + .thenReturn(defaultPath) + when(manager.sqlResolutionPathEntries(any[String], any[Seq[String]])) + .thenReturn(defaultPath) manager } @@ -230,6 +237,8 @@ class PlanResolutionSuite extends SharedSparkSession with AnalysisTest { invocation.getArguments()(0).asInstanceOf[String] match { case "testcat" => testCat + case CatalogManager.SESSION_CATALOG_NAME => + v2SessionCatalog case name => throw QueryExecutionErrors.catalogNotFoundError(name) } }) @@ -237,6 +246,14 @@ class PlanResolutionSuite extends SharedSparkSession with AnalysisTest { when(manager.currentNamespace).thenReturn(Array("default")) when(manager.v1SessionCatalog).thenReturn(v1SessionCatalog) when(manager.tempVariableManager).thenReturn(tempVariableManager) + when(manager.sessionPathEntries).thenReturn(None) + val defaultPath2 = SQLConf.get.resolutionSearchPath( + (v2SessionCatalog.name() +: Array("default")).toSeq) + when(manager.sqlResolutionPathEntries( + any[String], any[Seq[String]], any[String], any[Seq[String]])) + .thenReturn(defaultPath2) + when(manager.sqlResolutionPathEntries(any[String], any[Seq[String]])) + .thenReturn(defaultPath2) manager } From fe6051ad5fa404e0d55e44829f3e81b5d5609d87 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Tue, 28 Apr 2026 10:49:41 -0700 Subject: [PATCH 006/286] Revert "[SPARK-55952][SPARK-55953][SQL] Add ResolveChangelogTable analyzer rule for batch CDC post-processing" This reverts commit 881957a463e1fd32458688e8bcff6b74f95d2ec7. --- .../resources/error/error-conditions.json | 28 - .../sql/connector/catalog/Changelog.java | 9 - .../sql/catalyst/analysis/Analyzer.scala | 1 - .../analysis/ResolveChangelogTable.scala | 312 ----- .../sql/errors/QueryCompilationErrors.scala | 19 - .../datasources/v2/ChangelogTable.scala | 3 +- .../catalog/InMemoryChangelogCatalog.scala | 71 +- .../connector/ChangelogEndToEndSuite.scala | 26 +- .../connector/ChangelogResolutionSuite.scala | 76 +- ...lveChangelogTablePostProcessingSuite.scala | 1034 ----------------- 10 files changed, 30 insertions(+), 1549 deletions(-) delete mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala delete mode 100644 sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json index ff34214e2ad95..e6c786640e090 100644 --- a/common/utils/src/main/resources/error/error-conditions.json +++ b/common/utils/src/main/resources/error/error-conditions.json @@ -661,19 +661,6 @@ ], "sqlState" : "42P08" }, - "CHANGELOG_CONTRACT_VIOLATION" : { - "message" : [ - "The Change Data Capture (CDC) connector violated the `Changelog` contract at runtime." - ], - "subClass" : { - "UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION" : { - "message" : [ - "Connector emitted multiple delete or insert rows for the same `(rowId, _commit_version)` partition. The `Changelog` contract requires at most one logical change per row identity per commit when `containsIntermediateChanges() = false`. Either fix the connector to deduplicate intermediate states, or set `containsIntermediateChanges() = true` and use `deduplicationMode = netChanges`." - ] - } - }, - "sqlState" : "XX000" - }, "CHECKPOINT_FILE_CHECKSUM_VERIFICATION_FAILED" : { "message" : [ "Checksum verification failed, the file may be corrupted. File: ", @@ -3291,21 +3278,6 @@ "message" : [ "`startingVersion` is required when `endingVersion` is specified for CDC queries." ] - }, - "NET_CHANGES_NOT_YET_SUPPORTED" : { - "message" : [ - "The `deduplicationMode = netChanges` option on connector `` is not yet supported. Use `deduplicationMode = dropCarryovers` (default) or `deduplicationMode = none` instead." - ] - }, - "STREAMING_POST_PROCESSING_NOT_SUPPORTED" : { - "message" : [ - "Change Data Capture (CDC) streaming reads on connector `` do not yet support post-processing (carry-over removal, update detection, or net change computation). The requested combination of options would require post-processing, which is currently only available for batch reads. Use a batch read, or set `deduplicationMode = none` and `computeUpdates = false` to receive raw change rows in streaming." - ] - }, - "UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL" : { - "message" : [ - "`computeUpdates` cannot be used with `deduplicationMode=none` on connector `` because the connector emits copy-on-write carry-over pairs (`containsCarryoverRows()` returns true) that would be silently mislabeled as updates. Set `deduplicationMode` to `dropCarryovers` or `netChanges`." - ] } }, "sqlState" : "42K03" diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java index 5f2203aa1c379..0a811aa0ae4d7 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java @@ -43,15 +43,6 @@ @Evolving public interface Changelog { - /** Constant for the {@code _change_type} value of a row inserted into the table. */ - String CHANGE_TYPE_INSERT = "insert"; - /** Constant for the {@code _change_type} value of a row deleted from the table. */ - String CHANGE_TYPE_DELETE = "delete"; - /** Constant for the {@code _change_type} value of an update's pre-image row. */ - String CHANGE_TYPE_UPDATE_PREIMAGE = "update_preimage"; - /** Constant for the {@code _change_type} value of an update's post-image row. */ - String CHANGE_TYPE_UPDATE_POSTIMAGE = "update_postimage"; - /** A name to identify this changelog. */ String name(); diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala index 323a7db9c7ad7..a72824f953c08 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala @@ -466,7 +466,6 @@ class Analyzer( new ResolveCatalogs(catalogManager) :: ResolveInsertInto :: ResolveRelations :: - ResolveChangelogTable :: ResolvePartitionSpec :: ResolveFieldNameAndPosition :: AddMetadataColumns :: diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala deleted file mode 100644 index bdf9b9fed09cc..0000000000000 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala +++ /dev/null @@ -1,312 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.spark.sql.catalyst.analysis - -import org.apache.spark.sql.catalyst.expressions._ -import org.apache.spark.sql.catalyst.expressions.aggregate.{Count, Max, Min} -import org.apache.spark.sql.catalyst.plans.logical._ -import org.apache.spark.sql.catalyst.rules.Rule -import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 -import org.apache.spark.sql.connector.catalog.{Changelog, ChangelogInfo} -import org.apache.spark.sql.errors.QueryCompilationErrors -import org.apache.spark.sql.execution.datasources.v2.{ChangelogTable, DataSourceV2Relation} -import org.apache.spark.sql.types.{IntegerType, StringType} - -/** - * Post-processes a resolved [[ChangelogTable]] read to apply CDC option semantics - * (carry-over removal, update detection) and to enforce supported option combinations. - * - * Fires after [[ResolveRelations]] has wrapped the connector's [[Changelog]] in a - * [[ChangelogTable]]. Both batch ([[DataSourceV2Relation]]) and streaming - * ([[StreamingRelationV2]]) reads are handled: - * - Batch: the requested post-processing passes are injected as logical operators on top - * of the relation. Carry-over removal and update detection are fused into a single - * pass over a (rowId, _commit_version)-partitioned Window: the Filter drops CoW - * carry-over pairs (same rowVersion on both sides) and the subsequent Project relabels - * real delete+insert pairs as update_preimage / update_postimage. - * - Streaming: post-processing is not yet supported. If the requested options would - * require any post-processing, the rule throws an explicit [[AnalysisException]] to - * prevent silent wrong results. Streams that don't require post-processing pass - * through unchanged. - * - * Net change computation (`deduplicationMode = netChanges`) is not yet implemented and - * is rejected up-front for both batch and streaming. - */ -object ResolveChangelogTable extends Rule[LogicalPlan] { - - /** - * Reserved (`__spark_cdc_*`) column names used internally by post-processing; - * connectors must not emit columns with these names. - */ - object HelperColumn { - final val DelCnt = "__spark_cdc_del_cnt" - final val InsCnt = "__spark_cdc_ins_cnt" - final val MinRv = "__spark_cdc_min_rv" - final val MaxRv = "__spark_cdc_max_rv" - final val RvCnt = "__spark_cdc_rv_cnt" - - val all: Set[String] = Set(DelCnt, InsCnt, MinRv, MaxRv, RvCnt) - } - - override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp { - case rel @ DataSourceV2Relation(table: ChangelogTable, _, _, _, _, _) if !table.resolved => - val changelog = table.changelog - val req = evaluateRequirements(changelog, table.changelogInfo) - - val resolvedRel = rel.copy(table = table.copy(resolved = true)) - var updatedRel: LogicalPlan = resolvedRel - if (req.requiresCarryOverRemoval || req.requiresUpdateDetection) { - updatedRel = addRowLevelPostProcessing( - resolvedRel, changelog, req.requiresCarryOverRemoval, req.requiresUpdateDetection) - } - if (req.requiresNetChanges) { - updatedRel = injectNetChangeComputation(updatedRel, changelog) - } - updatedRel - - case rel @ StreamingRelationV2(_, _, table: ChangelogTable, _, _, _, _, _, _) - if !table.resolved => - // Streaming CDC reads do not yet apply post-processing. Run the same option / - // capability validation as the batch path so silent wrong results are impossible: - // either no post-processing would be required (fall through, return raw stream), - // or we throw an explicit AnalysisException. - val changelog = table.changelog - val req = evaluateRequirements(changelog, table.changelogInfo) - if (req.needsAny) { - throw QueryCompilationErrors.cdcStreamingPostProcessingNotSupported(changelog.name()) - } - rel.copy(table = table.copy(resolved = true)) - } - - // --------------------------------------------------------------------------- - // Option validation & Requirement Computation - // --------------------------------------------------------------------------- - - /** - * Captures which post-processing passes a CDC query requires, derived from the - * user-provided [[ChangelogInfo]] options and the connector-declared [[Changelog]] - * capability flags. - */ - private case class PostProcessingRequirements( - requiresCarryOverRemoval: Boolean, - requiresUpdateDetection: Boolean, - requiresNetChanges: Boolean) { - def needsAny: Boolean = - requiresCarryOverRemoval || requiresUpdateDetection || requiresNetChanges - } - - /** - * Validates CDC option/capability combinations and computes which post-processing - * passes are required. Throws an [[org.apache.spark.sql.AnalysisException]] for - * unsupported or contradictory combinations (currently: `netChanges` deduplication, - * and `computeUpdates` with surfaced carry-overs but no carry-over removal). - */ - private def evaluateRequirements( - changelog: Changelog, - options: ChangelogInfo): PostProcessingRequirements = { - // Net change computation is not yet implemented. - if (options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NET_CHANGES) { - throw QueryCompilationErrors.cdcNetChangesNotYetSupported(changelog.name()) - } - - val requiresCarryOverRemoval = - options.deduplicationMode() != ChangelogInfo.DeduplicationMode.NONE && - changelog.containsCarryoverRows() - val requiresUpdateDetection = - options.computeUpdates() && changelog.representsUpdateAsDeleteAndInsert() - val requiresNetChanges = - options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NET_CHANGES && - changelog.containsIntermediateChanges() - - // If carry-overs are surfaced and update detection is enabled without carry-over - // removal, carry-overs would be falsely classified as updates, leading to wrong - // results. Hence we throw. - if (requiresUpdateDetection && - changelog.containsCarryoverRows() && - options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NONE) { - throw QueryCompilationErrors.cdcUpdateDetectionRequiresCarryOverRemoval( - changelog.name()) - } - - PostProcessingRequirements( - requiresCarryOverRemoval, requiresUpdateDetection, requiresNetChanges) - } - - // --------------------------------------------------------------------------- - // Row Level Post Processing (Update Detection & Carry-over Removal) - // --------------------------------------------------------------------------- - - /** - * Adds row-level post-processing (carry-over removal and/or update detection) on top of - * the given plan. `counts` = per-partition delete and insert change_type row counts over - * `(rowId, _commit_version)`. `rv bounds` = per-partition min/max of `rowVersion`. - * Equal bounds signal a copy-on-write carry-over. - * - both active -> Window(counts + rv bounds) -> Filter -> Project(relabel) -> Drop helpers - * - carry-over only -> Window(counts + rv bounds) -> Filter -> Drop helpers - * - update only -> Window(counts only) -> Project(relabel) -> Drop helpers - * - neither -> not invoked (caller guards this case) - */ - private def addRowLevelPostProcessing( - plan: LogicalPlan, - cl: Changelog, - requiresCarryOverRemoval: Boolean, - requiresUpdateDetection: Boolean): LogicalPlan = { - // Row-version bounds in the Window are needed iff we filter carry-over pairs. - var modifiedPlan = addPostProcessingWindow(plan, cl, - includeRowVersionBounds = requiresCarryOverRemoval) - if (requiresCarryOverRemoval) modifiedPlan = addCarryOverPairFilter(modifiedPlan) - if (requiresUpdateDetection) modifiedPlan = addUpdateRelabelProjection(modifiedPlan) - removeHelperColumns(modifiedPlan) - } - - /** - * Adds a Window node partitioned by (rowId, _commit_version) that computes - * `_del_cnt` and `_ins_cnt` per partition, and, when `includeRowVersionBounds` - * is true, additionally `_min_rv` / `_max_rv` / `_rv_cnt` (min, max and non-null - * count of `Changelog.rowVersion()`). - * - * `_del_cnt` / `_ins_cnt` drive update detection (1 each -> relabel as - * update_preimage / update_postimage). `_min_rv` / `_max_rv` / `_rv_cnt` drive - * carry-over detection (within a delete+insert pair, `_rv_cnt = 2` AND equal - * bounds signal a CoW carry-over). - */ - private def addPostProcessingWindow( - plan: LogicalPlan, - cl: Changelog, - includeRowVersionBounds: Boolean): LogicalPlan = { - val changeTypeAttr = getAttribute(plan, "_change_type") - val rowIdExprs = V2ExpressionUtils.resolveRefs[NamedExpression](cl.rowId().toSeq, plan) - val commitVersionAttr = getAttribute(plan, "_commit_version") - val partitionByCols = rowIdExprs ++ Seq(commitVersionAttr) - val windowSpec = WindowSpecDefinition(partitionByCols, Nil, UnspecifiedFrame) - - val insertIf = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_INSERT)), - Literal(1), Literal(null, IntegerType)) - val deleteIf = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_DELETE)), - Literal(1), Literal(null, IntegerType)) - - val insCntAlias = Alias(WindowExpression( - Count(Seq(insertIf)).toAggregateExpression(), windowSpec), HelperColumn.InsCnt)() - val delCntAlias = Alias(WindowExpression( - Count(Seq(deleteIf)).toAggregateExpression(), windowSpec), HelperColumn.DelCnt)() - val baseAliases = Seq(delCntAlias, insCntAlias) - val rowVersionAliases = if (includeRowVersionBounds) { - val rowVersionExpr = - V2ExpressionUtils.resolveRef[NamedExpression](cl.rowVersion(), plan) - Seq( - Alias(WindowExpression( - Min(rowVersionExpr).toAggregateExpression(), windowSpec), HelperColumn.MinRv)(), - Alias(WindowExpression( - Max(rowVersionExpr).toAggregateExpression(), windowSpec), HelperColumn.MaxRv)(), - Alias(WindowExpression( - Count(Seq(rowVersionExpr)).toAggregateExpression(), windowSpec), HelperColumn.RvCnt)()) - } else { - Seq.empty - } - Window(baseAliases ++ rowVersionAliases, partitionByCols, Nil, plan) - } - - /** - * Adds a Filter node that drops rows belonging to a CoW carry-over pair. - * A pair is a carry-over iff - * `_del_cnt = 1 AND _ins_cnt = 1 AND _rv_cnt = 2 AND _min_rv = _max_rv`. - * The `_rv_cnt = 2` clause guards against a NULL rowVersion silently matching - * `_min_rv = _max_rv` (Spark's min/max skip NULLs). - */ - private def addCarryOverPairFilter(input: LogicalPlan): LogicalPlan = { - val delCnt = getAttribute(input, HelperColumn.DelCnt) - val insCnt = getAttribute(input, HelperColumn.InsCnt) - val minRv = getAttribute(input, HelperColumn.MinRv) - val maxRv = getAttribute(input, HelperColumn.MaxRv) - val rvCnt = getAttribute(input, HelperColumn.RvCnt) - - val isCarryoverPair = And( - And(EqualTo(delCnt, Literal(1L)), EqualTo(insCnt, Literal(1L))), - And(EqualTo(rvCnt, Literal(2L)), EqualTo(minRv, maxRv))) - Filter(Not(isCarryoverPair), input) - } - - /** - * Adds a Project node that rewrites `_change_type` to `update_preimage` / - * `update_postimage` whenever a delete+insert pair is present in the partition. - * Expects the input to expose `_del_cnt` and `_ins_cnt`. - */ - private def addUpdateRelabelProjection(input: LogicalPlan): LogicalPlan = { - val changeTypeAttr = getAttribute(input, "_change_type") - val delCnt = getAttribute(input, HelperColumn.DelCnt) - val insCnt = getAttribute(input, HelperColumn.InsCnt) - - val isUpdate = And( - EqualTo(delCnt, Literal(1L)), - EqualTo(insCnt, Literal(1L))) - val isInvalid = Or(GreaterThan(delCnt, Literal(1L)), GreaterThan(insCnt, Literal(1L))) - val updateType = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_INSERT)), - Literal(Changelog.CHANGE_TYPE_UPDATE_POSTIMAGE), - Literal(Changelog.CHANGE_TYPE_UPDATE_PREIMAGE)) - - val raiseInvalid = RaiseError( - Literal("CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION"), - CreateMap(Nil), - StringType) - val caseExpr = CaseWhen(Seq(isInvalid -> raiseInvalid, isUpdate -> updateType), changeTypeAttr) - - val projectList = input.output.map { attr => - if (attr.name == "_change_type") Alias(caseExpr, "_change_type")() - else attr - } - Project(projectList, input) - } - - // --------------------------------------------------------------------------- - // Net Change Computation - // --------------------------------------------------------------------------- - - /** - * Collapses multiple changes per row identity into the net effect. - * Not yet implemented. - */ - private def injectNetChangeComputation( - plan: LogicalPlan, - cl: Changelog): LogicalPlan = { - plan - } - - // --------------------------------------------------------------------------- - // Helpers - // --------------------------------------------------------------------------- - - /** - * Removes any helper columns (see [[HelperColumn]]) that earlier steps added to the - * plan. Helper columns not present in the input are silently ignored, so this method - * can be applied unconditionally regardless of which post-processing steps ran. - */ - private def removeHelperColumns(input: LogicalPlan): LogicalPlan = { - Project(input.output.filterNot(a => HelperColumn.all.contains(a.name)), input) - } - - /** - * Looks up an attribute by name in a plan's output. Throws a clear error if missing -- - * used for required columns like `_change_type` / `_commit_version` / helper columns - * added by earlier steps; a missing column is always a programming error. - */ - private def getAttribute(plan: LogicalPlan, name: String): Attribute = - plan.output.find(_.name == name).getOrElse( - throw new IllegalStateException( - s"Required column '$name' not found in plan output: " + - plan.output.map(_.name).mkString(", "))) -} diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala index 02e9f188e0fa4..b596d2f95391f 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala @@ -3862,25 +3862,6 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat messageParameters = Map("catalogName" -> catalogName)) } - def cdcUpdateDetectionRequiresCarryOverRemoval( - changelogName: String): AnalysisException = { - new AnalysisException( - errorClass = "INVALID_CDC_OPTION.UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL", - messageParameters = Map("changelogName" -> changelogName)) - } - - def cdcNetChangesNotYetSupported(changelogName: String): AnalysisException = { - new AnalysisException( - errorClass = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", - messageParameters = Map("changelogName" -> changelogName)) - } - - def cdcStreamingPostProcessingNotSupported(changelogName: String): AnalysisException = { - new AnalysisException( - errorClass = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", - messageParameters = Map("changelogName" -> changelogName)) - } - def invalidCdcOptionConflictingRangeTypes(): Throwable = { new AnalysisException( errorClass = "INVALID_CDC_OPTION.CONFLICTING_RANGE_TYPES", diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala index bb5a03f64990d..8521df3db2ff0 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala @@ -33,8 +33,7 @@ import org.apache.spark.sql.util.CaseInsensitiveStringMap */ case class ChangelogTable( changelog: Changelog, - changelogInfo: ChangelogInfo, - resolved: Boolean = false) extends Table with SupportsRead { + changelogInfo: ChangelogInfo) extends Table with SupportsRead { override def name: String = changelog.name diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala index 3a37b0a84fa26..c47ed2668e3b4 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala @@ -23,7 +23,6 @@ import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.analysis.NoSuchTableException import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.ChangelogRange.{TimestampRange, UnboundedRange, VersionRange} -import org.apache.spark.sql.connector.expressions.{FieldReference, NamedReference} import org.apache.spark.sql.connector.read._ import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, Offset} import org.apache.spark.sql.types._ @@ -45,22 +44,6 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { private var _lastChangelogInfo: Option[ChangelogInfo] = None def lastChangelogInfo: Option[ChangelogInfo] = _lastChangelogInfo - // Per-table overrides for Changelog properties (carry-over rows, intermediate changes, - // update representation, row identity). Tests can set these to exercise post-processing. - private val changelogProperties: mutable.Map[String, ChangelogProperties] = - mutable.Map.empty - - /** - * Override the [[Changelog]] properties returned for a given table. - * Defaults are: containsCarryoverRows=false, containsIntermediateChanges=false, - * representsUpdateAsDeleteAndInsert=false, no rowId, no rowVersion. - */ - def setChangelogProperties( - ident: Identifier, - properties: ChangelogProperties): Unit = { - changelogProperties(ident.toString) = properties - } - override def loadChangelog( ident: Identifier, changelogInfo: ChangelogInfo): Changelog = { @@ -75,9 +58,8 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { // _commit_version is at index numDataCols + 1 (after _change_type) val commitVersionIdx = numDataCols + 1 val filtered = filterByRange(allRows.toSeq, commitVersionIdx, changelogInfo.range()) - val props = changelogProperties.getOrElse(ident.toString, ChangelogProperties()) new InMemoryChangelog( - table.name + "_changelog", table.columns, filtered, props) + table.name + "_changelog", table.columns, filtered) } /** @@ -127,42 +109,15 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { } } -/** - * Configurable properties for [[InMemoryChangelog]] that test cases can use to exercise - * Spark's post-processing (carry-over removal, update detection, net changes). - * - * @param containsCarryoverRows whether the change stream may contain identical CoW pairs - * @param containsIntermediateChanges whether multiple changes per row may exist - * @param representsUpdateAsDeleteAndInsert whether updates appear as raw delete+insert - * @param rowIdNames optional row identity columns as top-level names (e.g. Seq("id")) - * @param rowIdPaths optional row identity paths for nested struct fields - * (e.g. Seq(Seq("payload", "id"))); takes precedence over rowIdNames - * @param rowVersionName optional row version column (e.g. Some("row_commit_version")); - * must be a per-row version that distinguishes carry-overs from - * real updates. Do NOT pass the commit version, which is constant - * within a partition and would cause every delete+insert pair to - * look like a carry-over - */ -case class ChangelogProperties( - containsCarryoverRows: Boolean = false, - containsIntermediateChanges: Boolean = false, - representsUpdateAsDeleteAndInsert: Boolean = false, - rowIdNames: Seq[String] = Seq.empty, - rowIdPaths: Seq[Seq[String]] = Seq.empty, - rowVersionName: Option[String] = None) - /** * A test [[Changelog]] that returns pre-populated change rows. * - * Properties (carry-over presence, update representation, row identity) are configurable - * via the [[ChangelogProperties]] parameter so tests can exercise different code paths - * in Spark's post-processing analyzer rule. + * Reports `containsCarryoverRows = false` so Spark skips carry-over removal. */ class InMemoryChangelog( tableName: String, dataColumns: Array[Column], - changeRows: Seq[InternalRow], - properties: ChangelogProperties = ChangelogProperties()) extends Changelog { + changeRows: Seq[InternalRow]) extends Changelog { private val cdcColumns: Array[Column] = dataColumns ++ Array( Column.create("_change_type", StringType), @@ -173,25 +128,11 @@ class InMemoryChangelog( override def columns(): Array[Column] = cdcColumns - override def containsCarryoverRows(): Boolean = properties.containsCarryoverRows - - override def containsIntermediateChanges(): Boolean = properties.containsIntermediateChanges + override def containsCarryoverRows(): Boolean = false - override def representsUpdateAsDeleteAndInsert(): Boolean = - properties.representsUpdateAsDeleteAndInsert + override def containsIntermediateChanges(): Boolean = false - override def rowId(): Array[NamedReference] = { - if (properties.rowIdPaths.nonEmpty) { - properties.rowIdPaths.map(parts => FieldReference(parts): NamedReference).toArray - } else { - properties.rowIdNames.map(name => FieldReference.column(name): NamedReference).toArray - } - } - - override def rowVersion(): NamedReference = properties.rowVersionName match { - case Some(name) => FieldReference.column(name) - case None => super.rowVersion() - } + override def representsUpdateAsDeleteAndInsert(): Boolean = false override def newScanBuilder( options: CaseInsensitiveStringMap): ScanBuilder = { diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala index 9622d23122318..006b645193023 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala @@ -418,22 +418,27 @@ class ChangelogEndToEndSuite extends SharedSparkSession { ChangelogInfo.DeduplicationMode.NONE) } - test("changes() passes computeUpdates to catalog") { + test("changes() passes deduplicationMode and computeUpdates to catalog") { catalog.addChangeRows(ident, Seq( makeChangeRow(1L, "a", "insert", 1L, 1000000L))) // DataFrame API spark.read .option("startingVersion", "1") + .option("deduplicationMode", "netChanges") .option("computeUpdates", "true") .changes(fullTableName) .collect() - assert(catalog.lastChangelogInfo.get.computeUpdates() === true) + val info1 = catalog.lastChangelogInfo.get + assert(info1.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) + assert(info1.computeUpdates() === true) // SQL sql(s"SELECT * FROM $fullTableName CHANGES FROM VERSION 1 " + - "WITH (computeUpdates = 'true')").collect() - assert(catalog.lastChangelogInfo.get.computeUpdates() === true) + "WITH (deduplicationMode = 'netChanges', computeUpdates = 'true')").collect() + val info2 = catalog.lastChangelogInfo.get + assert(info2.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) + assert(info2.computeUpdates() === true) } // ---------- Batch: timestamp range ---------- @@ -584,20 +589,23 @@ class ChangelogEndToEndSuite extends SharedSparkSession { // ---------- Streaming: CDC options ---------- - test("streaming changes() passes computeUpdates to catalog") { + test("streaming changes() passes deduplicationMode and computeUpdates to catalog") { catalog.addChangeRows(ident, Seq( makeChangeRow(1L, "a", "insert", 1L, 1000000L))) // DataFrame API val dfApiStream = spark.readStream .option("startingVersion", "1") + .option("deduplicationMode", "netChanges") .option("computeUpdates", "true") .changes(fullTableName) val q1 = dfApiStream.writeStream .format("memory").queryName("cdc_stream_opts_df").start() try { q1.processAllAvailable() - assert(catalog.lastChangelogInfo.get.computeUpdates() === true) + val info1 = catalog.lastChangelogInfo.get + assert(info1.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) + assert(info1.computeUpdates() === true) } finally { q1.stop() } @@ -605,12 +613,14 @@ class ChangelogEndToEndSuite extends SharedSparkSession { // SQL val sqlStream = sql( s"SELECT * FROM STREAM $fullTableName CHANGES FROM VERSION 1 " + - "WITH (computeUpdates = 'true')") + "WITH (deduplicationMode = 'netChanges', computeUpdates = 'true')") val q2 = sqlStream.writeStream .format("memory").queryName("cdc_stream_opts_sql").start() try { q2.processAllAvailable() - assert(catalog.lastChangelogInfo.get.computeUpdates() === true) + val info2 = catalog.lastChangelogInfo.get + assert(info2.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) + assert(info2.computeUpdates() === true) } finally { q2.stop() } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala index d403db1e62bf9..db6817b0c212c 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala @@ -17,14 +17,13 @@ package org.apache.spark.sql.connector -import java.util.Collections +import java.util import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 import org.apache.spark.sql.connector.catalog._ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.ChangelogRange -import org.apache.spark.sql.connector.expressions.Transform import org.apache.spark.sql.execution.datasources.v2.{ChangelogTable, DataSourceV2Relation} import org.apache.spark.sql.test.SharedSparkSession import org.apache.spark.sql.types.{LongType, StringType} @@ -64,8 +63,8 @@ class ChangelogResolutionSuite extends SharedSparkSession { Array( Column.create("id", LongType), Column.create("data", StringType)), - Array.empty[Transform], - Collections.emptyMap[String, String]()) + Array.empty, + new util.HashMap[String, String]()) val noCdcCat = spark.sessionState.catalogManager.catalog(noCdcCatalogName).asTableCatalog val ident2 = Identifier.of(Array.empty, "test_table") @@ -77,8 +76,8 @@ class ChangelogResolutionSuite extends SharedSparkSession { Array( Column.create("id", LongType), Column.create("data", StringType)), - Array.empty[Transform], - Collections.emptyMap[String, String]()) + Array.empty, + new util.HashMap[String, String]()) } test("CHANGES clause resolves to DataSourceV2Relation with ChangelogTable") { @@ -204,69 +203,4 @@ class ChangelogResolutionSuite extends SharedSparkSession { assert(range.startingVersion() == "1") assert(range.endingVersion().get() == "5") } - - // =========================================================================== - // Streaming post-processing rejection - // =========================================================================== - // - // Streaming CDC reads bypass the post-processing analyzer rule's transformation - // path. To prevent silent wrong results when the requested options would require - // post-processing, the rule throws an explicit AnalysisException for streaming. - - /** Re-creates the test table with non-nullable columns suitable as rowId / rowVersion. */ - private def recreatePostProcessingTable(): Identifier = { - val cat = spark.sessionState.catalogManager.catalog(cdcCatalogName).asTableCatalog - val ident = Identifier.of(Array.empty, "test_table") - if (cat.tableExists(ident)) cat.dropTable(ident) - cat.createTable( - ident, - Array( - Column.create("id", LongType, false), - Column.create("row_commit_version", LongType, false)), - Array.empty[Transform], - Collections.emptyMap[String, String]()) - ident - } - - test("DataStreamReader - changes() with carry-over capability throws") { - val ident = recreatePostProcessingTable() - val cat = spark.sessionState.catalogManager - .catalog(cdcCatalogName) - .asInstanceOf[InMemoryChangelogCatalog] - cat.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - checkError( - intercept[AnalysisException] { - spark.readStream - .changes(s"$cdcCatalogName.test_table") - .queryExecution.analyzed - }, - condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", - parameters = Map("changelogName" -> s"$cdcCatalogName.test_table_changelog")) - } - - test("DataStreamReader - changes() with computeUpdates throws") { - val ident = recreatePostProcessingTable() - val cat = spark.sessionState.catalogManager - .catalog(cdcCatalogName) - .asInstanceOf[InMemoryChangelogCatalog] - cat.setChangelogProperties(ident, ChangelogProperties( - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - checkError( - intercept[AnalysisException] { - spark.readStream - .option("computeUpdates", "true") - .option("deduplicationMode", "none") - .changes(s"$cdcCatalogName.test_table") - .queryExecution.analyzed - }, - condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", - parameters = Map("changelogName" -> s"$cdcCatalogName.test_table_changelog")) - } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala deleted file mode 100644 index 353472a035f91..0000000000000 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala +++ /dev/null @@ -1,1034 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.spark.sql.connector - -import java.util.Collections - -import org.scalatest.BeforeAndAfterEach - -import org.apache.spark.SparkRuntimeException -import org.apache.spark.sql.{AnalysisException, QueryTest, Row} -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 -import org.apache.spark.sql.connector.catalog.{ - ChangelogProperties, Column, Identifier, InMemoryChangelogCatalog} -import org.apache.spark.sql.connector.catalog.Changelog.{ - CHANGE_TYPE_DELETE, CHANGE_TYPE_INSERT, CHANGE_TYPE_UPDATE_POSTIMAGE, - CHANGE_TYPE_UPDATE_PREIMAGE} -import org.apache.spark.sql.connector.expressions.Transform -import org.apache.spark.sql.execution.datasources.v2.ChangelogTable -import org.apache.spark.sql.test.SharedSparkSession -import org.apache.spark.sql.types.{ - BinaryType, BooleanType, DoubleType, LongType, StringType, StructField, StructType} -import org.apache.spark.unsafe.types.UTF8String - -/** - * Tests for [[org.apache.spark.sql.catalyst.analysis.ResolveChangelogTable]] using the - * in-memory changelog catalog. These tests don't depend on Delta or any specific connector; - * they directly control what the connector "returns" by populating the in-memory changelog - * with hand-crafted change rows. - * - * Each test sets up [[ChangelogProperties]] on the catalog to enable specific post-processing - * paths (carry-over removal, update detection) and then verifies that Spark's analyzer rule - * correctly transforms the plan and produces the expected output. - */ -class ResolveChangelogTablePostProcessingSuite - extends QueryTest - with SharedSparkSession - with BeforeAndAfterEach { - - private val catalogName = "cdc_test_catalog" - private val testTableName = "events" - - override def beforeAll(): Unit = { - super.beforeAll() - spark.conf.set( - s"spark.sql.catalog.$catalogName", - classOf[InMemoryChangelogCatalog].getName) - } - - override def beforeEach(): Unit = { - super.beforeEach() - val cat = catalog - val ident = Identifier.of(Array.empty, testTableName) - if (cat.tableExists(ident)) cat.dropTable(ident) - cat.clearChangeRows(ident) - cat.setChangelogProperties(ident, ChangelogProperties()) - cat.createTable( - ident, - Array( - Column.create("id", LongType), - Column.create("name", StringType), - Column.create("row_commit_version", LongType, false)), - Array.empty[Transform], - Collections.emptyMap[String, String]()) - } - - private def catalog: InMemoryChangelogCatalog = { - spark.sessionState.catalogManager - .catalog(catalogName) - .asInstanceOf[InMemoryChangelogCatalog] - } - - private def ident = Identifier.of(Array.empty, testTableName) - - /** - * Helper to create a change row matching schema - * (id, name, row_commit_version, _change_type, _commit_version, _commit_timestamp). - * - * `rowCommitVersion` follows Delta row-tracking semantics: carry-over pairs (CoW-rewritten - * unchanged rows) share the same value on both sides; real updates carry the OLD value on - * the delete side and the NEW value on the insert side. Defaults to `commitVersion` for - * tests that don't exercise carry-over removal. - */ - private def changeRow( - id: Long, - name: String, - changeType: String, - commitVersion: Long, - rowCommitVersion: Long = -1L, - commitTimestamp: Long = 0L): InternalRow = { - val rcv = if (rowCommitVersion == -1L) commitVersion else rowCommitVersion - InternalRow( - id, - UTF8String.fromString(name), - rcv, - UTF8String.fromString(changeType), - commitVersion, - commitTimestamp) - } - - // =========================================================================== - // Carry-Over Removal - // =========================================================================== - - test("carry-over removal drops identical delete+insert pairs") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - // v1: insert Alice and Bob (rcv=1 each) - // v2: real delete Alice (preimage carries old rcv=1); - // carry-over for Bob (CoW, rcv unchanged on both sides) - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // carry-over - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) // carry-over (same rcv) - - checkAnswer( - sql( - s"SELECT id, name, _change_type, _commit_version " + - s"FROM $catalogName.$testTableName CHANGES FROM VERSION 1 TO VERSION 2"), - Seq( - Row(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - Row(2L, "Bob", CHANGE_TYPE_INSERT, 1L), - Row(1L, "Alice", CHANGE_TYPE_DELETE, 2L))) - } - - test("deduplicationMode=none keeps all carry-over rows") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) - - checkAnswer( - sql( - s"SELECT id FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (deduplicationMode = 'none')"), - Seq(Row(1L), Row(2L), Row(2L))) - } - - test("NULL rowVersion on one side is NOT silently dropped as carry-over") { - // Regression for a NULL-safety hole: min/max skip NULLs, so _min_rv = _max_rv alone - // would match a pair with one NULL and one non-null rowVersion. The _rv_cnt = 2 - // clause in the carry-over filter prevents that. - // - // The fixture table here declares `row_commit_version` as nullable so the optimizer - // is not allowed to fold IsNull(non-nullable-col) to false; the NULL is a legitimate - // value the guard must defend against. - val nullableRcvTable = "events_nullable_rcv" - val nullableIdent = Identifier.of(Array.empty, nullableRcvTable) - val cat = catalog - if (cat.tableExists(nullableIdent)) cat.dropTable(nullableIdent) - cat.clearChangeRows(nullableIdent) - cat.createTable( - nullableIdent, - Array( - Column.create("id", LongType), - Column.create("name", StringType), - Column.create("row_commit_version", LongType, true)), - Array.empty[Transform], - Collections.emptyMap[String, String]()) - cat.setChangelogProperties(nullableIdent, ChangelogProperties( - containsCarryoverRows = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - cat.addChangeRows(nullableIdent, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - // v2: one side has NULL rowVersion (buggy connector), the other has a real value. - InternalRow(1L, UTF8String.fromString("Alice"), null, - UTF8String.fromString(CHANGE_TYPE_DELETE), 2L, 0L), - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 5L))) - - checkAnswer( - sql(s"SELECT id, name, _change_type, _commit_version " + - s"FROM $catalogName.$nullableRcvTable CHANGES FROM VERSION 1 TO VERSION 2"), - Seq( - Row(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - Row(1L, "Alice", CHANGE_TYPE_DELETE, 2L), - Row(1L, "Alice", CHANGE_TYPE_INSERT, 2L))) - } - - // =========================================================================== - // Update Detection - // =========================================================================== - - test("update detection relabels delete+insert with different data as update") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = false, // no carry-overs in this test - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - // v2: Alice -> Robert (delete old, insert new) - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), - changeRow(1L, "Robert", CHANGE_TYPE_INSERT, 2L))) - - val rows = sql( - s"SELECT id, name, _change_type, _commit_version " + - s"FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") - .orderBy("_commit_version", "_change_type") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") - - assert(descs.contains("1:Alice:insert"), s"v1 insert. Got: ${descs.mkString(",")}") - assert(descs.contains("1:Alice:update_preimage")) - assert(descs.contains("1:Robert:update_postimage")) - // No raw delete/insert at v2 - assert(!descs.contains("1:Alice:delete")) - assert(!descs.contains("1:Robert:insert")) - } - - test("delete and insert in different versions are NOT labeled as update") { - catalog.setChangelogProperties(ident, ChangelogProperties( - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 3L))) - - val rows = sql( - s"SELECT _change_type, _commit_version FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 3 " + - s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") - .collect() - - assert(!rows.exists(_.getString(0).contains("update_")), - "Delete and insert in different versions should not be labeled as update") - } - - // =========================================================================== - // Composite rowId: partitioning uses every rowId column - // =========================================================================== - // - // With a composite rowId such as Seq("id", "name"), the (rowId, _commit_version) - // window partition must include BOTH columns. A regression that drops one of the - // rowId columns would either falsely merge two different row identities into one - // partition (silently mislabeling unrelated delete/insert pairs as updates) or - // trip the UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION runtime guard. - - test("update detection with composite rowId keeps different (id, name) tuples raw") { - catalog.setChangelogProperties(ident, ChangelogProperties( - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id", "name"), - rowVersionName = Some("row_commit_version"))) - - // delete (1, Alice) and insert (1, Bob) at v2. These are DIFFERENT composite - // rowIds; they must NOT be relabeled as update. - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), - changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 2L))) - - val rows = sql( - s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}").toSet - - assert(descs == Set("1:Alice:delete", "1:Bob:insert"), - s"Composite rowId must keep different (id, name) tuples raw. Got: $descs") - } - - test("carry-over removal with composite rowId removes pairs per (id, name) tuple") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - rowIdNames = Seq("id", "name"), - rowVersionName = Some("row_commit_version"))) - - // Two independent carry-over pairs at v2, both with id=1 but different names. - // With correct composite-rowId partitioning, each pair lives in its own - // (id, name, _commit_version) partition, has _del_cnt=1 / _ins_cnt=1 and equal - // _min_rv / _max_rv, and gets dropped. With broken (id-only) partitioning, the - // four rows would collapse into one partition with _del_cnt=2 / _ins_cnt=2 and - // the carry-over filter (which requires =1) would keep them all. - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), - changeRow(1L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) - - val rows = sql( - s"SELECT id, name, _change_type, _commit_version " + - s"FROM $catalogName.$testTableName CHANGES FROM VERSION 2 TO VERSION 2") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") - assert(rows.isEmpty, - s"Both Alice and Bob carry-over pairs at v2 should be removed. Got: ${descs.mkString(",")}") - } - - // =========================================================================== - // No row identity: post-processing skipped - // =========================================================================== - - test("no capability flags -> post-processing not injected in plan") { - // Default ChangelogProperties has no capability flags set; the rule sees nothing to do. - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L))) - - val df = sql( - s"SELECT * FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") - - val plan = df.queryExecution.analyzed.treeString - assert(!plan.contains("__spark_cdc_del_cnt"), - s"Plan must not contain post-processing window helpers. Plan:\n$plan") - assert(!plan.contains("__spark_cdc_ins_cnt"), - s"Plan must not contain post-processing window helpers. Plan:\n$plan") - } - - test("streaming without post-processing options passes through") { - // Streaming reads with no capability flags on the connector and no - // post-processing options must resolve without the rule throwing. - val df = spark.readStream - .option("startingVersion", "1") - .changes(s"$catalogName.$testTableName") - val analyzed = df.queryExecution.analyzed - val plan = analyzed.treeString - assert(!plan.contains("__spark_cdc_del_cnt"), - s"Streaming plan must not contain post-processing helpers. Plan:\n$plan") - - // Positive assertion: the rule actually fired on the streaming relation. Without this, - // a regression that deletes the streaming arm of `ResolveChangelogTable.apply` would - // also pass the absence-of-helpers check above. - val tableResolved = analyzed.collectFirst { - case rel: StreamingRelationV2 if rel.table.isInstanceOf[ChangelogTable] => - rel.table.asInstanceOf[ChangelogTable].resolved - } - assert(tableResolved.contains(true), - s"Expected ChangelogTable to be marked resolved by the rule. Plan:\n$plan") - } - - test("streaming with post-processing options is rejected") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - checkError( - exception = intercept[AnalysisException] { - spark.readStream - .option("startingVersion", "1") - .changes(s"$catalogName.$testTableName") - .queryExecution.analyzed - }, - condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", - parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) - } - - // =========================================================================== - // Combined - // =========================================================================== - - test("carry-over removal and update detection combined") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - // v1: insert Alice (rcv=1), Bob (rcv=1) - // v2: Alice carry-over (CoW, rcv unchanged), Bob real update (old rcv=1, new rcv=2) - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // carry-over - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), // carry-over - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // update preimage - changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) // update postimage - - val rows = sql( - s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") - .orderBy("_commit_version", "id", "_change_type") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}").toSet - - // v1 inserts - assert(descs.contains("1:Alice:insert")) - assert(descs.contains("2:Bob:insert")) - // Alice carry-over dropped - assert(!descs.contains("1:Alice:delete")) - // Bob -> Robert as update - assert(descs.contains("2:Bob:update_preimage")) - assert(descs.contains("2:Robert:update_postimage")) - // Should be exactly 4 rows - assert(rows.length == 4, s"Expected 4 rows, got ${rows.length}: ${descs.mkString(",")}") - } - - // =========================================================================== - // computeUpdates default (false) keeps raw delete+insert - // =========================================================================== - - test("without computeUpdates, delete+insert with different data stays raw") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - // Alice: carry-over (CoW, rcv unchanged on both sides) - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), - // Bob -> Robert: real change (old rcv on pre, new rcv on post) - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) - - // Default computeUpdates=false: do NOT relabel, but DO drop carry-overs - val rows = sql( - s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2") - .orderBy("_commit_version", "id", "_change_type") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") - - assert(descs.contains("2:Bob:delete"), s"Bob delete remains raw. Got: ${descs.mkString(",")}") - assert(descs.contains("2:Robert:insert"), "Robert insert remains raw") - assert(!descs.exists(_.contains("update_")), "No update_* without computeUpdates") - assert(!descs.contains("1:Alice:delete"), "Alice carry-over removed") - } - - test("update detection on pure inserts leaves them as inserts") { - catalog.setChangelogProperties(ident, ChangelogProperties( - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L))) - - val rows = sql( - s"SELECT id, _change_type FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") - .collect() - - assert(rows.length == 2) - assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT), - s"Pure inserts must stay 'insert'. Got: ${rows.map(_.getString(1)).mkString(",")}") - } - - // =========================================================================== - // Keep Carry-over Rows and deduplication flag tests - // =========================================================================== - - test("computeUpdates with deduplicationMode=none is rejected on COW connector") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - checkError( - intercept[AnalysisException] { - sql(s"SELECT * FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 " + - s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") - }, - condition = "INVALID_CDC_OPTION.UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL", - parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) - } - - test("computeUpdates with deduplicationMode=none is allowed on non-COW connector") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = false, // MOR-style: no carry-overs possible - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - // v2: Alice -> Robert (delete old, insert new) - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), - changeRow(1L, "Robert", CHANGE_TYPE_INSERT, 2L))) - - val rows = sql( - s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 " + - s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") - assert(descs.contains("1:Alice:update_preimage"), - s"Expected Alice update_preimage. Got: ${descs.mkString(",")}") - assert(descs.contains("1:Robert:update_postimage"), - s"Expected Robert update_postimage. Got: ${descs.mkString(",")}") - } - - // =========================================================================== - // Contract enforcement: at most one delete + one insert per (rowId, version) - // =========================================================================== - // - // With `representsUpdateAsDeleteAndInsert = true` and `containsIntermediateChanges = false`, - // the `Changelog` contract guarantees at most one logical change per (rowId, _commit_version) - // partition. The update-relabel projection enforces this at runtime: if it sees more than one - // delete or more than one insert in a partition, it raises - // INVALID_CDC_OPTION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION instead of silently - // mislabeling extra rows as updates. - - test("update detection raises on multiple inserts for same (rowId, _commit_version)") { - catalog.setChangelogProperties(ident, ChangelogProperties( - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - // Contract violation: 2 inserts for id=1 at v2. - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), - changeRow(1L, "Alice2", CHANGE_TYPE_INSERT, 2L), - changeRow(1L, "Alice3", CHANGE_TYPE_INSERT, 2L))) - - checkError( - intercept[SparkRuntimeException] { - sql(s"SELECT * FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") - .collect() - }, - condition = "CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION", - parameters = Map.empty) - } - - test("update detection raises on multiple deletes for same (rowId, _commit_version)") { - catalog.setChangelogProperties(ident, ChangelogProperties( - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - // Contract violation: 2 deletes for id=1 at v2. - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), - changeRow(1L, "Alice2", CHANGE_TYPE_DELETE, 2L), - changeRow(1L, "Alice3", CHANGE_TYPE_INSERT, 2L))) - - checkError( - intercept[SparkRuntimeException] { - sql(s"SELECT * FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") - .collect() - }, - condition = "CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION", - parameters = Map.empty) - } - - // =========================================================================== - // Net changes deduplication: not yet supported - // =========================================================================== - // - // `deduplicationMode = netChanges` collapses multiple changes per row identity into the - // net effect. It is not yet implemented in [[ResolveChangelogTable]]. - - test("deduplicationMode=netChanges is rejected when connector emits intermediate changes") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsIntermediateChanges = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - checkError( - intercept[AnalysisException] { - sql(s"SELECT * FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 " + - s"WITH (deduplicationMode = 'netChanges')") - }, - condition = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", - parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) - } - - test("deduplicationMode=netChanges is rejected even when connector has no intermediate changes") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsIntermediateChanges = false, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - checkError( - intercept[AnalysisException] { - sql(s"SELECT * FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 " + - s"WITH (deduplicationMode = 'netChanges')") - }, - condition = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", - parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) - } - - // =========================================================================== - // Range edge cases - // =========================================================================== - - test("multiple operations across versions") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - catalog.addChangeRows(ident, Seq( - // v1: insert 3 rows (rcv=1 each) - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - // v2: delete Alice (preimage carries old rcv=1); CoW carry-overs for Bob/Charlie - // keep rcv=1 on both sides (row unchanged). - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), - changeRow(3L, "Charlie", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), - // v3: update Bob -> Robert (old rcv=1, new rcv=3); CoW carry-over for Charlie (rcv=1) - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 3L, rowCommitVersion = 1L), - changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 3L, rowCommitVersion = 3L), - changeRow(3L, "Charlie", CHANGE_TYPE_DELETE, 3L, rowCommitVersion = 1L), - changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L, rowCommitVersion = 1L), - // v4: insert Diana (rcv=4) - changeRow(4L, "Diana", CHANGE_TYPE_INSERT, 4L, rowCommitVersion = 4L))) - - val rows = sql( - s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 4 WITH (computeUpdates = 'true')") - .orderBy("_commit_version", "id", "_change_type") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}").toSet - - // v1 - assert(descs.contains("1:Alice:insert:v1")) - assert(descs.contains("2:Bob:insert:v1")) - assert(descs.contains("3:Charlie:insert:v1")) - // v2 - assert(descs.contains("1:Alice:delete:v2")) - assert(!descs.contains("2:Bob:delete:v2"), "Bob carry-over dropped") - assert(!descs.contains("3:Charlie:delete:v2"), "Charlie carry-over dropped") - // v3 - assert(descs.contains("2:Bob:update_preimage:v3")) - assert(descs.contains("2:Robert:update_postimage:v3")) - assert(!descs.contains("3:Charlie:delete:v3"), "Charlie carry-over dropped in v3") - // v4 - assert(descs.contains("4:Diana:insert:v4")) - } - - test("larger insert batch returns all rows") { - catalog.addChangeRows(ident, (1 to 5).map(i => - changeRow(i.toLong, ('A' + i - 1).toChar.toString, CHANGE_TYPE_INSERT, 1L))) - - val rows = sql( - s"SELECT id, _change_type FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 1 WITH (deduplicationMode = 'none')") - .collect() - - assert(rows.length == 5) - assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT)) - } - - test("DELETE all rows: no carry-over inserts at v2") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - // v1 inserts carry rcv=1; v2 deletes carry the old rcv=1 (rcv tracks last modification) - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L))) - - val rows = sql( - s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2") - .orderBy("_commit_version", "id") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") - - assert(descs.contains("1:Alice:insert:v1")) - assert(descs.contains("2:Bob:insert:v1")) - assert(descs.contains("1:Alice:delete:v2")) - assert(descs.contains("2:Bob:delete:v2")) - assert(!descs.exists(_.contains("insert:v2")), "No inserts at v2") - } - - test("UPDATE all rows: every row gets update_pre/postimage, no carry-overs") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - // Every v2 row is a real update: delete side carries old rcv=1, insert side new rcv=2. - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(1L, "Alice_updated", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L), - changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(2L, "Bob_updated", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) - - val rows = sql( - s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") - .orderBy("_commit_version", "id", "_change_type") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}").toSet - - assert(descs.contains("1:Alice:update_preimage:v2")) - assert(descs.contains("1:Alice_updated:update_postimage:v2")) - assert(descs.contains("2:Bob:update_preimage:v2")) - assert(descs.contains("2:Bob_updated:update_postimage:v2")) - assert(rows.length == 6, s"Expected 2 inserts + 2 pre + 2 post. Got ${rows.length}") - } - - test("append-only workload: all inserts, no carry-over needed") { - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), - changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) - - val rows = sql( - s"SELECT id, _change_type FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 3") - .collect() - - assert(rows.length == 3) - assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT)) - } - - test("carry-over removal with many rows: only real change remains") { - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - // 10 inserts at v1 (rcv=1 each). At v2: delete row 5; CoW writes 9 carry-over pairs - // (rcv unchanged since v1, i.e. rcv=1 on both sides) plus 1 real delete (rcv=1, old). - val v1Inserts = (1 to 10).map(i => - changeRow( - i.toLong, ('A' + i - 1).toChar.toString, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L)) - val v2Carryovers = (1 to 10).filter(_ != 5).flatMap { i => - val name = ('A' + i - 1).toChar.toString - Seq( - changeRow(i.toLong, name, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(i.toLong, name, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L)) - } - val v2RealDelete = Seq(changeRow(5L, "E", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L)) - catalog.addChangeRows(ident, v1Inserts ++ v2Carryovers ++ v2RealDelete) - - val rows = sql( - s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 2 TO VERSION 2") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") - assert(rows.length == 1, - s"Only 1 real change should remain (9 carry-overs dropped). Got: ${descs.mkString(",")}") - assert(descs.contains("5:E:delete")) - } - - test("carry-over removal with mixed types (DOUBLE, BOOLEAN, BINARY)") { - val mixedTable = "events_mixed" - val mixedIdent = Identifier.of(Array.empty, mixedTable) - val cat = catalog - if (cat.tableExists(mixedIdent)) cat.dropTable(mixedIdent) - cat.clearChangeRows(mixedIdent) - cat.createTable( - mixedIdent, - Array( - Column.create("id", LongType), - Column.create("name", StringType), - Column.create("score", DoubleType), - Column.create("active", BooleanType), - Column.create("payload", BinaryType), - Column.create("row_commit_version", LongType, false)), - Array.empty[Transform], - Collections.emptyMap[String, String]()) - cat.setChangelogProperties(mixedIdent, ChangelogProperties( - containsCarryoverRows = true, - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - def mixedRow( - id: Long, name: String, score: Double, active: Boolean, payload: Array[Byte], - ct: String, v: Long, rowCommitVersion: Long): InternalRow = { - InternalRow( - id, UTF8String.fromString(name), score, active, payload, rowCommitVersion, - UTF8String.fromString(ct), v, 0L) - } - - val alicePayload = Array[Byte](1, 2, 3) - val bobPayload = Array[Byte](4, 5, 6) - - cat.addChangeRows(mixedIdent, Seq( - mixedRow( - 1L, "Alice", 95.5, true, alicePayload, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - mixedRow( - 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - // v2: update Alice's score (old rcv=1, new rcv=2); Bob is carry-over (rcv unchanged) - mixedRow( - 1L, "Alice", 95.5, true, alicePayload, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - mixedRow( - 1L, "Alice", 99.0, true, alicePayload, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L), - mixedRow( - 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - mixedRow( - 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) - - val rows = sql( - s"SELECT id, name, score, active, _change_type FROM $catalogName.$mixedTable " + - s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") - .orderBy("_commit_version", "id", "_change_type") - .collect() - - val descs = rows.map(r => s"${r.getLong(0)}:${r.getString(4)}") - assert(descs.contains("1:update_preimage")) - assert(descs.contains("1:update_postimage")) - assert(!descs.contains("2:delete"), - s"Bob carry-over must be dropped despite DOUBLE/BOOLEAN/BINARY. Got: " + - descs.mkString(",")) - - val pre = rows.find(r => - r.getLong(0) == 1L && r.getString(4) == CHANGE_TYPE_UPDATE_PREIMAGE).get - val post = rows.find(r => - r.getLong(0) == 1L && r.getString(4) == CHANGE_TYPE_UPDATE_POSTIMAGE).get - assert(pre.getDouble(2) == 95.5) - assert(post.getDouble(2) == 99.0) - } - - // =========================================================================== - // Regression: nested rowId + nested rowVersion end-to-end - // =========================================================================== - - // End-to-end check that nested rowId paths (e.g. `payload.id`) are resolved on the plan - // and threaded through carry-over detection. The pair survives the filter because the - // row_commit_version differs across delete/insert, not because of any sibling-field data. - test("nested rowId path resolves correctly through carry-over filter") { - val nestedTable = "events_nested" - val nestedIdent = Identifier.of(Array.empty, nestedTable) - val cat = catalog - if (cat.tableExists(nestedIdent)) cat.dropTable(nestedIdent) - cat.clearChangeRows(nestedIdent) - - val payloadType = StructType(Seq( - StructField("id", LongType), - StructField("value", StringType))) - - cat.createTable( - nestedIdent, - Array( - Column.create("payload", payloadType), - Column.create("row_commit_version", LongType, false)), - Array.empty[Transform], - Collections.emptyMap[String, String]()) - - cat.setChangelogProperties(nestedIdent, ChangelogProperties( - containsCarryoverRows = true, - rowIdPaths = Seq(Seq("payload", "id")), - rowVersionName = Some("row_commit_version"))) - - def nestedRow( - id: Long, value: String, ct: String, v: Long, rowCommitVersion: Long): InternalRow = { - InternalRow( - InternalRow(id, UTF8String.fromString(value)), - rowCommitVersion, - UTF8String.fromString(ct), v, 0L) - } - - cat.addChangeRows(nestedIdent, Seq( - nestedRow(1L, "original", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - // v2 update: rowId same, rowVersion differs (old rcv=1 on preimage, new rcv=2 on postimage) - nestedRow(1L, "original", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - nestedRow(1L, "CHANGED", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) - - val rows = sql( - s"SELECT payload.id AS id, payload.value AS value, _change_type, _commit_version " + - s"FROM $catalogName.$nestedTable CHANGES FROM VERSION 1 TO VERSION 2") - .orderBy("_commit_version", "_change_type") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") - - assert(descs.contains("1:original:insert:v1"), - s"v1 insert must survive. Got: ${descs.mkString(",")}") - assert(descs.contains("1:original:delete:v2"), - s"v2 delete must survive (payload.value differs from insert). Got: ${descs.mkString(",")}") - assert(descs.contains("1:CHANGED:insert:v2"), - s"v2 insert must survive (payload.value differs from delete). Got: ${descs.mkString(",")}") - assert(rows.length == 3, - s"Expected 3 rows (v1 insert + v2 delete + v2 insert). Got ${rows.length}: " + - descs.mkString(",")) - } - - // =========================================================================== - // No-op UPDATE is correctly preserved as update_preimage/postimage - // =========================================================================== - - test("no-op UPDATE is labeled as update (row_commit_version differs on pre/post)") { - // A no-op UPDATE bumps row_commit_version even when data is byte-identical, so the - // delete side carries the OLD rcv and the insert side the NEW rcv. Window post-processing - // sees different rowVersions, treats this as a real change, and labels both rows as - // update_preimage / update_postimage. - catalog.setChangelogProperties(ident, ChangelogProperties( - containsCarryoverRows = true, - representsUpdateAsDeleteAndInsert = true, - rowIdNames = Seq("id"), - rowVersionName = Some("row_commit_version"))) - - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), - // v2 no-op update: identical data, but rcv differs (Delta bumps it on any UPDATE) - changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) - - val rows = sql( - s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") - .orderBy("_commit_version", "_change_type") - .collect() - - val descs = rows.map(r => - s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") - - assert(descs.contains("1:Alice:insert:v1")) - assert(descs.contains("1:Alice:update_preimage:v2"), - s"No-op UPDATE preimage must be labeled. Got: ${descs.mkString(",")}") - assert(descs.contains("1:Alice:update_postimage:v2"), - s"No-op UPDATE postimage must be labeled. Got: ${descs.mkString(",")}") - assert(rows.length == 3, - s"Expected v1 insert + v2 update pre/post = 3 rows. Got ${rows.length}") - } - - // =========================================================================== - // Baseline (range syntax / connector range filtering -- rule bypassed via - // deduplicationMode = 'none'; included as smoke tests for the SQL surface). - // =========================================================================== - - test("baseline: single-version range FROM VERSION X TO VERSION X") { - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L), - changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 2L))) - - val rows = sql( - s"SELECT id, _change_type, _commit_version FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (deduplicationMode = 'none')") - .collect() - - assert(rows.length == 1, s"Single version: 1 row. Got ${rows.length}") - assert(rows(0).getLong(0) == 3L) - assert(rows(0).getString(1) == CHANGE_TYPE_INSERT) - } - - test("baseline: EXCLUSIVE start bound skips the start version") { - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), - changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) - - val rows = sql( - s"SELECT id, _commit_version FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 EXCLUSIVE TO VERSION 3 " + - s"WITH (deduplicationMode = 'none')") - .orderBy("_commit_version") - .collect() - - assert(!rows.exists(_.getLong(1) == 1L), "v1 must be excluded") - assert(rows.exists(_.getLong(0) == 2L), "Bob (v2) included") - assert(rows.exists(_.getLong(0) == 3L), "Charlie (v3) included") - } - - test("baseline: open-ended range (no TO clause) reads to latest") { - catalog.addChangeRows(ident, Seq( - changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), - changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), - changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) - - val rows = sql( - s"SELECT id, _commit_version FROM $catalogName.$testTableName " + - s"CHANGES FROM VERSION 1 WITH (deduplicationMode = 'none')") - .orderBy("_commit_version", "id") - .collect() - - assert(rows.length == 3, s"Open-ended range should see all 3. Got ${rows.length}") - assert(rows.exists(r => r.getLong(0) == 3L && r.getLong(1) == 3L)) - } -} From dbfde5ba81eb3f1eb5957ff5798d28983128367f Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Tue, 28 Apr 2026 11:05:18 -0700 Subject: [PATCH 007/286] [SPARK-56646][K8S][DOCS] Document K8s executor resize and recovery mode configs ### What changes were proposed in this pull request? This PR documents K8s executor resize and recovery mode configs for Apache Spark 4.2.0. - `spark.kubernetes.executor.resizeInterval` - `spark.kubernetes.executor.resizeThreshold` - `spark.kubernetes.executor.resizeFactor` - `spark.kubernetes.allocation.recoveryMode.enabled` The three `executor.resize*` configurations are consumed by `org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin`. ### Why are the changes needed? To improve the documentation. ### Does this PR introduce _any_ user-facing change? No. Documentation-only update. ### How was this patch tested? Manual verification: ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes #55584 from dongjoon-hyun/SPARK-56646. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- docs/running-on-kubernetes.md | 42 +++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index 777d8d4228e40..5aa906f9f37d9 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -708,6 +708,18 @@ See the [configuration page](configuration.html) for information on Spark config + + + + + + @@ -1747,6 +1759,36 @@ See the [configuration page](configuration.html) for information on Spark config + + + + + + + + + + + + + + + + + +
4.1.0
spark.kubernetes.allocation.recoveryMode.enabled(none) + When Spark driver detects an executor termination due to OOM, Spark starts to + allocate the recovery-mode executors which accept only a single task per executor JVM. + In other words, the recovery-mode executors replace the OOM-terminated executors to + survive from the resource-hungry tasks for the remaining tasks and stages. + If set to false, Spark will not use the recovery-mode executors. + 4.2.0
spark.kubernetes.jars.avoidDownloadSchemes (none) 3.3.0
spark.kubernetes.executor.resizeInterval0s + Interval between executor resize operations. To disable, set 0 (default). + Takes effect only when org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin + is registered via spark.plugins. + 4.2.0
spark.kubernetes.executor.resizeThreshold0.9 + The threshold to resize. + Takes effect only when org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin + is registered via spark.plugins. + 4.2.0
spark.kubernetes.executor.resizeFactor0.1 + The factor to resize. + Takes effect only when org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin + is registered via spark.plugins. + 4.2.0
#### Pod template properties From 46831d5e3b88a6c51752ed55bf7538e6808f674c Mon Sep 17 00:00:00 2001 From: Peter Toth Date: Tue, 28 Apr 2026 13:22:38 -0700 Subject: [PATCH 008/286] [SPARK-56549][SQL] Dynamically enable k-way merge in `GroupPartitionsExec` only when parent requires ordering MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? `GroupPartitionsExec` can coalesce multiple input partitions sharing the same partition key into one output partition. When `spark.sql.sources.v2.bucketing. preserveOrderingOnCoalesce.enabled` is set, it uses `SortedMergeCoalescedRDD` (k-way merge) to preserve the child's output ordering through coalescing. Previously the k-way merge path was taken unconditionally whenever the config was enabled and the child was safe, even when the parent operator did not require any ordering — incurring unnecessary merge overhead for e.g. hash joins. This PR introduces a dynamic gate: - A new `enableSortedMerge: Boolean` parameter on `GroupPartitionsExec` (default `false`) controls whether k-way merge is used. - `supportsColumnar()` is now true again when the config is on but `enableSortedMerge=false` (e.g., hash-join parent). - A new `tryEnableSortedMerge()` method returns `Some(copy(enableSortedMerge = true))` when feasible, `None` otherwise. - `EnsureRequirements`, when it would otherwise insert a `SortExec` above a subtree, first calls `tryEnableSortedMerge(child)`, which uses `multiTransformDownWithPruning` to lazily enumerate all alternative plans where one or more `GroupPartitionsExec` nodes have sorted merge enabled. Traversal is pruned at `SortExec` (explicit check) and nodes without `KeyedPartitioning` output partitioning (via `hasKeyedPartitioning`). The call site takes the first alternative whose `outputOrdering` satisfies the parent's requirement; if none exists, a `SortExec` is inserted as before. - A `hasCoalescing` lazy val is extracted to avoid repeating `groupedPartitions.exists(_._2.size > 1)`. ### Why are the changes needed? K-way merge is more expensive than simple sequential concatenation. When the parent operator does not require ordering (e.g. a hash join), this overhead is wasted. The dynamic gate ensures k-way merge is activated only when it actually satisfies a parent ordering requirement. ### Does this PR introduce _any_ user-facing change? No. But on the unreleased `master` branch when `spark.sql.sources.v2.bucketing.preserveOrderingOnCoalesce.enabled=true`, k-way merge is now applied selectively (only when a parent requires the ordering) rather than eagerly for all coalescing `GroupPartitionsExec` nodes. Operators that do not require ordering (e.g. hash joins) will use the cheaper `CoalescedRDD` path even with the config enabled. ### How was this patch tested? - `GroupPartitionsExecSuite`: updated existing SPARK-55715 tests to set `enableSortedMerge = true` explicitly; added new SPARK-56549 tests for `tryEnableSortedMerge()` covering all feasibility conditions. - `EnsureRequirementsSuite`: added new SPARK-56549 tests for `EnsureRequirements.tryEnableSortedMerge` covering traversal through plain unary nodes, binary nodes propagating ordering from one child, binary nodes with `PartitioningCollection` (both children have `KeyedPartitioning`), and pruning at `SortExec` and nodes without `KeyedPartitioning` output partitioning. - `KeyGroupedPartitioningSuite`: added end-to-end test asserting that a hash join keeps `enableSortedMerge = false` / uses `CoalescedRDD`, while a sort-merge join gets `enableSortedMerge = true` / uses `SortedMergeCoalescedRDD`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 4.6 Closes #55448 from peter-toth/SPARK-56549-dynamically-keep-outputOrdering. Authored-by: Peter Toth Signed-off-by: Dongjoon Hyun --- .../datasources/v2/GroupPartitionsExec.scala | 30 +++- .../exchange/EnsureRequirements.scala | 40 ++++- .../KeyGroupedPartitioningSuite.scala | 77 +++++++- .../v2/GroupPartitionsExecSuite.scala | 80 +++++++-- .../exchange/EnsureRequirementsSuite.scala | 170 +++++++++++++++++- 5 files changed, 371 insertions(+), 26 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala index 64c937499f742..264a0e954936f 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala @@ -50,13 +50,18 @@ import org.apache.spark.sql.vectorized.ColumnarBatch * @param distributePartitions When true, splits for a key are distributed across the expected * partitions (padding with empty partitions). When false, all splits * are replicated to every expected partition for that key. + * @param enableSortedMerge When true, uses [[SortedMergeCoalescedRDD]] to perform a k-way merge + * of the coalesced partitions, preserving the child's output ordering + * end-to-end. Set by [[EnsureRequirements]] when a parent operator + * requires the ordering that this node can satisfy via sorted merge. */ case class GroupPartitionsExec( child: SparkPlan, @transient joinKeyPositions: Option[Seq[Int]] = None, @transient expectedPartitionKeys: Option[Seq[(InternalRowComparableWrapper, Int)]] = None, @transient reducers: Option[Seq[Option[Reducer[_, _]]]] = None, - @transient distributePartitions: Boolean = false + @transient distributePartitions: Boolean = false, + @transient enableSortedMerge: Boolean = false ) extends UnaryExecNode { override def outputPartitioning: Partitioning = { @@ -160,6 +165,8 @@ case class GroupPartitionsExec( @transient lazy val isGrouped: Boolean = groupedPartitionsTuple._2 + @transient private lazy val hasCoalescing: Boolean = groupedPartitions.exists(_._2.size > 1) + // Whether the child subtree is safe to use with SortedMergeCoalescedRDD (k-way merge). // // --- The general problem --- @@ -223,10 +230,23 @@ case class GroupPartitionsExec( child.outputOrdering.nonEmpty && childIsSafeForKWayMerge + /** + * Returns a copy of this node with k-way merge enabled if it is feasible: the config is on, + * the child has an ordering, the child subtree is `SafeForKWayMerge`, and this node actually + * coalesces partitions. + */ + def tryEnableSortedMerge(): Option[GroupPartitionsExec] = { + Option.when(hasCoalescing && canUseSortedMerge) { + val newGroupPartitions = copy(enableSortedMerge = true) + newGroupPartitions.copyTagsFrom(this) + newGroupPartitions + } + } + override protected def doExecute(): RDD[InternalRow] = { if (groupedPartitions.isEmpty) { sparkContext.emptyRDD - } else if (canUseSortedMerge && groupedPartitions.exists(_._2.size > 1)) { + } else if (hasCoalescing && enableSortedMerge && canUseSortedMerge) { val partitionCoalescer = new GroupedPartitionCoalescer(groupedPartitions.map(_._2)) val rowOrdering = new LazyCodeGenOrdering(child.outputOrdering, child.output) new SortedMergeCoalescedRDD[InternalRow]( @@ -241,7 +261,7 @@ case class GroupPartitionsExec( } override def supportsColumnar: Boolean = - child.supportsColumnar && !(canUseSortedMerge && groupedPartitions.exists(_._2.size > 1)) + child.supportsColumnar && !(hasCoalescing && enableSortedMerge && canUseSortedMerge) override protected def doExecuteColumnar(): RDD[ColumnarBatch] = { if (groupedPartitions.isEmpty) { @@ -258,12 +278,12 @@ case class GroupPartitionsExec( copy(child = newChild) override def outputOrdering: Seq[SortOrder] = { - if (groupedPartitions.forall(_._2.size <= 1)) { + if (!hasCoalescing) { // No coalescing: each output partition is exactly one input partition. The child's // within-partition ordering is fully preserved (including any key-derived ordering that // `DataSourceV2ScanExecBase` already prepended). child.outputOrdering - } else if (canUseSortedMerge) { + } else if (enableSortedMerge && canUseSortedMerge) { // Coalescing with sorted merge: SortedMergeCoalescedRDD performs a k-way merge using the // child's ordering, so the full within-partition ordering is preserved end-to-end. child.outputOrdering diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala index 1fff6c22c5ad8..62a3a977162aa 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala @@ -273,7 +273,7 @@ case class EnsureRequirements( child match { case ShuffleExchangeExec(_, c, so, ps) => ShuffleExchangeExec(newPartitioning, c, so, ps) - case GroupPartitionsExec(c, _, _, _, _) => ShuffleExchangeExec(newPartitioning, c) + case gpe: GroupPartitionsExec => ShuffleExchangeExec(newPartitioning, gpe.child) case _ => ShuffleExchangeExec(newPartitioning, child) } } @@ -286,13 +286,49 @@ case class EnsureRequirements( if (SortOrder.orderingSatisfies(child.outputOrdering, requiredOrdering)) { child } else { - SortExec(requiredOrdering, global = false, child = child) + // Before adding a SortExec, check whether a GroupPartitionsExec anywhere in the child + // subtree can self-satisfy via sorted merge. tryEnableSortedMerge generates all alternative + // plans where one or more GPEs have sorted merge enabled; we take the first one whose + // outputOrdering satisfies the requirement. + tryEnableSortedMerge(child) + .find(newChild => SortOrder.orderingSatisfies(newChild.outputOrdering, requiredOrdering)) + .getOrElse(SortExec(requiredOrdering, global = false, child = child)) } } children } + private def hasKeyedPartitioning(p: Partitioning): Boolean = p match { + case e: Expression => e.exists(_.isInstanceOf[KeyedPartitioning]) + case _ => false + } + + // Generates all alternative plans in which one or more GroupPartitionsExec nodes in the subtree + // have sorted-merge enabled (every possible combination). Returns a LazyList so the caller can + // stop evaluating once a satisfying alternative is found. + // + // Pruning: traversal stops at SortExec (which reorders data, making sorted merge below it + // pointless) and at any node whose outputPartitioning no longer carries a KeyedPartitioning. + // This is a good heuristic, though not strictly equivalent to "ordering no longer propagates": + // partition-key expressions are constant within each coalesced partition and therefore usually + // prefix outputOrdering. When a node prunes the KeyedPartitioning (e.g. a Project that drops + // partition keys), it also prunes that ordering prefix. Since Spark has no notion of constant + // expressions in SortOrder, dropping a prefix invalidates the rest of the ordering too -- so in + // practice the two are always pruned together. + // + // At each GPE the rule emits [original, sorted-merge-enabled] alternatives (or just [original] + // when sorted merge cannot be enabled). multiTransformDownWithPruning then builds the Cartesian + // product across all GPEs in the subtree, giving every combination. + private[exchange] def tryEnableSortedMerge(plan: SparkPlan): LazyList[SparkPlan] = + plan.multiTransformDownWithPruning( + p => !p.isInstanceOf[SortExec] && + hasKeyedPartitioning(p.asInstanceOf[SparkPlan].outputPartitioning)) { + case gpe: GroupPartitionsExec => + // Include the original so that peer GPEs are still independently considered. + gpe +: gpe.tryEnableSortedMerge().toSeq + } + private def reorder( leftKeys: IndexedSeq[Expression], rightKeys: IndexedSeq[Expression], diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala index 4a406322a5a19..38de6b043bc2b 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala @@ -33,7 +33,7 @@ import org.apache.spark.sql.connector.expressions.Expressions._ import org.apache.spark.sql.execution.{ExtendedMode, FormattedMode, RDDScanExec, SimpleMode, SortExec, SparkPlan} import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, DataSourceV2ScanRelation, GroupPartitionsExec} import org.apache.spark.sql.execution.exchange.{ShuffleExchangeExec, ShuffleExchangeLike} -import org.apache.spark.sql.execution.joins.SortMergeJoinExec +import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, SortMergeJoinExec} import org.apache.spark.sql.functions.{col, max} import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.internal.SQLConf._ @@ -3913,4 +3913,79 @@ class KeyGroupedPartitioningSuite extends DistributionAndOrderingSuiteBase with } } } + + test("SPARK-56549: k-way merge enabled only when parent requires ordering") { + // Both tables are partitioned by id/item_id and report a two-column ordering. + // Key 1 appears on two splits on each side, so GroupPartitionsExec must coalesce. + // + // Dynamic gate: with the config enabled, k-way merge must be activated only when the parent + // actually requires ordering (SMJ), and must stay off when the parent does not (hash join). + val itemOrdering = Array( + sort(FieldReference("id"), SortDirection.ASCENDING, NullOrdering.NULLS_FIRST), + sort(FieldReference("arrive_time"), SortDirection.ASCENDING, NullOrdering.NULLS_FIRST)) + createTable(items, itemsColumns, Array(identity("id")), itemOrdering) + sql(s"INSERT INTO testcat.ns.$items VALUES " + + "(2, 'cc', 30.0, cast('2023-06-15' as timestamp)), " + + "(1, 'bb', 20.0, cast('2022-03-10' as timestamp)), " + + "(3, 'dd', 40.0, cast('2024-01-01' as timestamp)), " + + "(1, 'aa', 10.0, cast('2021-05-20' as timestamp)), " + + "(2, 'ee', 50.0, cast('2025-09-01' as timestamp))") + + val purchaseOrdering = Array( + sort(FieldReference("item_id"), SortDirection.ASCENDING, NullOrdering.NULLS_FIRST), + sort(FieldReference("time"), SortDirection.ASCENDING, NullOrdering.NULLS_FIRST)) + createTable(purchases, purchasesColumns, Array(identity("item_id")), purchaseOrdering) + sql(s"INSERT INTO testcat.ns.$purchases VALUES " + + "(2, 50.0, cast('2025-09-01' as timestamp)), " + + "(1, 10.0, cast('2021-05-20' as timestamp)), " + + "(3, 40.0, cast('2024-01-01' as timestamp)), " + + "(2, 30.0, cast('2023-06-15' as timestamp)), " + + "(1, 20.0, cast('2022-03-10' as timestamp))") + + withSQLConf( + SQLConf.REQUIRE_ALL_CLUSTER_KEYS_FOR_CO_PARTITION.key -> "false", + SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true" + ) { + val hashDf = sql( + s""" + |SELECT /*+ SHUFFLE_HASH(i, p) */ i.id, i.name + |FROM testcat.ns.$items i + |JOIN testcat.ns.$purchases p ON p.item_id = i.id AND p.time = i.arrive_time + |""".stripMargin) + checkAnswer(hashDf, Seq(Row(1, "aa"), Row(1, "bb"), Row(2, "cc"), Row(2, "ee"), Row(3, "dd"))) + val hashPlan = hashDf.queryExecution.executedPlan + assert(collect(hashPlan) { case j: ShuffledHashJoinExec => j }.nonEmpty, + "expected ShuffledHashJoinExec") + assert(collectAllShuffles(hashPlan).isEmpty, "should not shuffle for compatible partitioning") + val hashCoalescing = + collectAllGroupPartitions(hashPlan).filter(_.groupedPartitions.exists(_._2.size > 1)) + assert(hashCoalescing.nonEmpty, "expected coalescing GroupPartitionsExec") + hashCoalescing.foreach { gp => + assert(!gp.enableSortedMerge, + "hash join does not require ordering: enableSortedMerge must stay false") + assert(!gp.execute().isInstanceOf[SortedMergeCoalescedRDD[_]], + "hash join does not require ordering: must use simple CoalescedRDD") + } + + val smjDf = sql( + s""" + |${selectWithMergeJoinHint("i", "p")} + |i.id, i.name + |FROM testcat.ns.$items i + |JOIN testcat.ns.$purchases p ON p.item_id = i.id AND p.time = i.arrive_time + |""".stripMargin) + checkAnswer(smjDf, Seq(Row(1, "aa"), Row(1, "bb"), Row(2, "cc"), Row(2, "ee"), Row(3, "dd"))) + val smjPlan = smjDf.queryExecution.executedPlan + assert(collectAllShuffles(smjPlan).isEmpty, "should not shuffle for compatible partitioning") + val smjCoalescing = + collectAllGroupPartitions(smjPlan).filter(_.groupedPartitions.exists(_._2.size > 1)) + assert(smjCoalescing.nonEmpty, "expected coalescing GroupPartitionsExec") + smjCoalescing.foreach { gp => + assert(gp.enableSortedMerge, + "sort-merge join requires ordering: enableSortedMerge must be true") + assert(gp.execute().isInstanceOf[SortedMergeCoalescedRDD[_]], + "sort-merge join requires ordering: must use SortedMergeCoalescedRDD") + } + } + } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExecSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExecSuite.scala index 35b3a32e33d82..5d2adeb0c00af 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExecSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExecSuite.scala @@ -146,7 +146,7 @@ class GroupPartitionsExecSuite extends SharedSparkSession { test("SPARK-55715: sorted merge config enabled but child not SafeForKWayMerge falls back " + "to key-expression ordering") { // DummySparkPlan does not extend SafeForKWayMerge, so childIsSafeForKWayMerge = false and - // canUseSortedMerge = false even when the preserve-ordering config is on. outputOrdering must + // canUseSortedMerge = false even with enableSortedMerge = true. outputOrdering must // therefore fall back to key-expression filtering (not return the full child ordering). val partitionKeys = Seq(row(1), row(2), row(1)) val childOrdering = Seq(SortOrder(exprA, Ascending), SortOrder(exprC, Ascending)) @@ -159,19 +159,19 @@ class GroupPartitionsExecSuite extends SharedSparkSession { withSQLConf( SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true", SQLConf.V2_BUCKETING_PRESERVE_KEY_ORDERING_ON_COALESCE_ENABLED.key -> "true") { - // Even though preserve-ordering is enabled, the child is not safe for k-way merge, + // Even though enableSortedMerge = true, the child is not safe for k-way merge, // so only key-expression orders survive (non-key exprC is dropped). - val ordering = GroupPartitionsExec(child).outputOrdering + val ordering = GroupPartitionsExec(child, enableSortedMerge = true).outputOrdering assert(ordering.length === 1) assert(ordering.head.child === exprA) } } - test("SPARK-55715: coalescing with sorted merge config enabled returns full child ordering") { - // Key 1 appears on partitions 0 and 2, causing coalescing. The child is a LeafExecNode - // so childIsSafeForKWayMerge = true. With the preserve-ordering config enabled, case 2 - // of outputOrdering kicks in and the full child ordering (including the non-key exprC) must - // be returned, not just the subset of key-expression orders. + test("SPARK-55715: coalescing with enableSortedMerge = true returns full child ordering") { + // Key 1 appears on partitions 0 and 2, causing coalescing. The child is a LeafExecNode so + // childIsSafeForKWayMerge = true. With enableSortedMerge = true and the config enabled, + // canUseSortedMerge = true and the full child ordering (including the non-key exprC) must be + // returned, not just the subset of key-expression orders. val partitionKeys = Seq(row(1), row(2), row(1)) val childOrdering = Seq(SortOrder(exprA, Ascending), SortOrder(exprC, Ascending)) val child = DummyLeafSparkPlan( @@ -181,19 +181,77 @@ class GroupPartitionsExecSuite extends SharedSparkSession { assert(!GroupPartitionsExec(child).groupedPartitions.forall(_._2.size <= 1), "expected coalescing") withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true") { - // Config enabled: k-way merge preserves full ordering including non-key exprC. - assert(GroupPartitionsExec(child).outputOrdering === childOrdering) + assert(GroupPartitionsExec(child).outputOrdering !== childOrdering, + "config alone should not enable k-way merge; enableSortedMerge must be set by planner") + assert(GroupPartitionsExec(child, enableSortedMerge = true).outputOrdering === childOrdering) } withSQLConf( SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "false", SQLConf.V2_BUCKETING_PRESERVE_KEY_ORDERING_ON_COALESCE_ENABLED.key -> "true") { // Sorted-merge config disabled, key-ordering config enabled: only key-expression orders // survive simple concatenation (non-key exprC is dropped). - val ordering = GroupPartitionsExec(child).outputOrdering + val ordering = GroupPartitionsExec(child, enableSortedMerge = true).outputOrdering assert(ordering.length === 1) assert(ordering.head.child === exprA) } } + + test("SPARK-56549: tryEnableSortedMerge returns Some when conditions are met") { + val partitionKeys = Seq(row(1), row(2), row(1)) + val childOrdering = Seq(SortOrder(exprA, Ascending), SortOrder(exprC, Ascending)) + val child = DummyLeafSparkPlan( + outputPartitioning = KeyedPartitioning(Seq(exprA), partitionKeys), + outputOrdering = childOrdering) + val gpe = GroupPartitionsExec(child) + + withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true") { + val result = gpe.tryEnableSortedMerge() + assert(result.isDefined) + assert(result.get.enableSortedMerge) + assert(result.get.outputOrdering === childOrdering) + } + } + + test("SPARK-56549: tryEnableSortedMerge returns None when config is disabled") { + val partitionKeys = Seq(row(1), row(2), row(1)) + val childOrdering = Seq(SortOrder(exprA, Ascending)) + val child = DummyLeafSparkPlan( + outputPartitioning = KeyedPartitioning(Seq(exprA), partitionKeys), + outputOrdering = childOrdering) + val gpe = GroupPartitionsExec(child) + + withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "false") { + assert(gpe.tryEnableSortedMerge().isEmpty) + } + } + + test("SPARK-56549: tryEnableSortedMerge returns None when child is not SafeForKWayMerge") { + val partitionKeys = Seq(row(1), row(2), row(1)) + val childOrdering = Seq(SortOrder(exprA, Ascending)) + // DummySparkPlan does not extend SafeForKWayMerge + val child = DummySparkPlan( + outputPartitioning = KeyedPartitioning(Seq(exprA), partitionKeys), + outputOrdering = childOrdering) + val gpe = GroupPartitionsExec(child) + + withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true") { + assert(gpe.tryEnableSortedMerge().isEmpty) + } + } + + test("SPARK-56549: tryEnableSortedMerge returns None when no coalescing occurs") { + val partitionKeys = Seq(row(1), row(2), row(3)) + val childOrdering = Seq(SortOrder(exprA, Ascending)) + val child = DummyLeafSparkPlan( + outputPartitioning = KeyedPartitioning(Seq(exprA), partitionKeys), + outputOrdering = childOrdering) + val gpe = GroupPartitionsExec(child) + + assert(gpe.groupedPartitions.forall(_._2.size <= 1), "expected non-coalescing") + withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true") { + assert(gpe.tryEnableSortedMerge().isEmpty) + } + } } private case class DummyLeafSparkPlan( diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala index 9c67a334c801c..2ff4652646cca 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala @@ -18,6 +18,7 @@ package org.apache.spark.sql.execution.exchange import org.apache.spark.api.python.PythonEvalType +import org.apache.spark.rdd.RDD import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.catalyst.expressions.DirectShufflePartitionID @@ -27,7 +28,7 @@ import org.apache.spark.sql.catalyst.plans.Inner import org.apache.spark.sql.catalyst.plans.physical.{SinglePartition, _} import org.apache.spark.sql.catalyst.statsEstimation.StatsTestPlan import org.apache.spark.sql.connector.catalog.functions._ -import org.apache.spark.sql.execution.{DummySparkPlan, SortExec} +import org.apache.spark.sql.execution.{BinaryExecNode, DummySparkPlan, LeafExecNode, SafeForKWayMerge, SortExec, UnaryExecNode} import org.apache.spark.sql.execution.SparkPlan import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, GroupPartitionsExec} import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, SortMergeJoinExec} @@ -1037,10 +1038,10 @@ class EnsureRequirementsSuite extends SharedSparkSession { case SortMergeJoinExec(_, _, _, _, SortExec(_, _, GroupPartitionsExec(DummySparkPlan(_, _, left: KeyedPartitioning, _, _), - _, _, _, _), _), + _, _, _, _, _), _), SortExec(_, _, GroupPartitionsExec(DummySparkPlan(_, _, right: KeyedPartitioning, _, _), - _, _, _, _), _), + _, _, _, _, _), _), _) => assert(left.expressions === Seq(bucket(4, exprB), bucket(8, exprC))) assert(right.expressions === Seq(bucket(4, exprC), bucket(8, exprB))) @@ -1061,10 +1062,10 @@ class EnsureRequirementsSuite extends SharedSparkSession { case SortMergeJoinExec(_, _, _, _, SortExec(_, _, GroupPartitionsExec(DummySparkPlan(_, _, left: PartitioningCollection, _, _), - _, _, _, _), _), + _, _, _, _, _), _), SortExec(_, _, GroupPartitionsExec(DummySparkPlan(_, _, right: KeyedPartitioning, _, _), - _, _, _, _), _), + _, _, _, _, _), _), _) => assert(left.partitionings.length == 2) assert(left.partitionings.head.isInstanceOf[KeyedPartitioning]) @@ -1096,10 +1097,10 @@ class EnsureRequirementsSuite extends SharedSparkSession { case SortMergeJoinExec(_, _, _, _, SortExec(_, _, GroupPartitionsExec(DummySparkPlan(_, _, left: PartitioningCollection, _, _), - _, _, _, _), _), + _, _, _, _, _), _), SortExec(_, _, GroupPartitionsExec(DummySparkPlan(_, _, right: PartitioningCollection, _, _), - _, _, _, _), _), + _, _, _, _, _), _), _) => assert(left.partitionings.length == 2) assert(left.partitionings.head.isInstanceOf[KeyedPartitioning]) @@ -1397,4 +1398,159 @@ class EnsureRequirementsSuite extends SharedSparkSession { requiredChildDistribution = Seq(UnspecifiedDistribution), requiredChildOrdering = Seq(Seq.empty) ) + + test("SPARK-56549: tryEnableSortedMerge traversal continues through plain unary nodes") { + withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true") { + val exprKey = AttributeReference("k", IntegerType)() + val partitionKeys = Seq(InternalRow(1), InternalRow(2), InternalRow(1)) + val ordering = Seq(SortOrder(exprKey, Ascending)) + val leaf = DummyLeafSafeForKWayMerge( + outputPartitioning = KeyedPartitioning(Seq(exprKey), partitionKeys), + outputOrdering = ordering) + val gpe = GroupPartitionsExec(leaf) + + // Baseline: GPE at root -- at least one alternative has sorted merge enabled. + assert(EnsureRequirements.tryEnableSortedMerge(gpe).exists(anyGpeEnabled)) + // Plain unary wrapper (e.g. FilterExec): traversal continues and sorted merge is enabled. + assert(EnsureRequirements.tryEnableSortedMerge(DummyPassthroughUnaryExec(gpe)) + .exists(anyGpeEnabled)) + // Two levels of plain unary wrappers: still enabled. + assert(EnsureRequirements.tryEnableSortedMerge( + DummyPassthroughUnaryExec(DummyPassthroughUnaryExec(gpe))) + .exists(anyGpeEnabled)) + } + } + + test("SPARK-56549: tryEnableSortedMerge traversal continues through binary nodes that " + + "propagate ordering from one child (e.g. ShuffledHashJoinExec stream side)") { + withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true") { + val exprKey = AttributeReference("k", IntegerType)() + val partitionKeys = Seq(InternalRow(1), InternalRow(2), InternalRow(1)) + val ordering = Seq(SortOrder(exprKey, Ascending)) + val leaf = DummyLeafSafeForKWayMerge( + outputPartitioning = KeyedPartitioning(Seq(exprKey), partitionKeys), + outputOrdering = ordering) + val gpe = GroupPartitionsExec(leaf) + val otherChild = DummyLeafSafeForKWayMerge() + + // Binary node whose ordering comes from left child (GPE side): sorted merge enabled. + assert(EnsureRequirements.tryEnableSortedMerge(DummyOrderFromLeftBinaryExec(gpe, otherChild)) + .exists(anyGpeEnabled)) + // Binary node with GPE only on the non-ordering (right) side: the binary node's + // outputPartitioning = left.outputPartitioning carries no KeyedPartitioning, so the pruning + // condition stops traversal at the root; no GPE is enabled. + assert(!EnsureRequirements.tryEnableSortedMerge(DummyOrderFromLeftBinaryExec(otherChild, gpe)) + .exists(anyGpeEnabled)) + } + } + + test("SPARK-56549: tryEnableSortedMerge traversal through binary nodes with " + + "PartitioningCollection (KP from both children, e.g. SHJ InnerLike)") { + withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true") { + val exprKey = AttributeReference("k", IntegerType)() + val partitionKeys = Seq(InternalRow(1), InternalRow(2), InternalRow(1)) + val ordering = Seq(SortOrder(exprKey, Ascending)) + val leaf = DummyLeafSafeForKWayMerge( + outputPartitioning = KeyedPartitioning(Seq(exprKey), partitionKeys), + outputOrdering = ordering) + val gpe = GroupPartitionsExec(leaf) + val otherChild = DummyLeafSafeForKWayMerge( + outputPartitioning = UnknownPartitioning(gpe.outputPartitioning.numPartitions)) + + // GPE on ordering (left) side: sorted merge is enabled and the binary's outputOrdering + // becomes non-empty. + assert(EnsureRequirements.tryEnableSortedMerge(DummyBothKPBinaryExec(gpe, otherChild)) + .exists(p => anyGpeEnabled(p) && p.outputOrdering.nonEmpty)) + + // GPE on non-ordering (right) side: the PartitioningCollection on the binary node includes + // KP from the right child, so traversal enters the binary and sorted merge IS enabled on the + // GPE. However, the binary's outputOrdering remains empty: it comes from the left (non-GPE) + // child. The call site's find correctly rejects all such alternatives. + assert(EnsureRequirements.tryEnableSortedMerge(DummyBothKPBinaryExec(otherChild, gpe)) + .exists(anyGpeEnabled)) + assert(!EnsureRequirements.tryEnableSortedMerge(DummyBothKPBinaryExec(otherChild, gpe)) + .exists(_.outputOrdering.nonEmpty)) + } + } + + test("SPARK-56549: tryEnableSortedMerge traversal stops at SortExec and Exchange") { + withSQLConf(SQLConf.V2_BUCKETING_PRESERVE_ORDERING_ON_COALESCE_ENABLED.key -> "true") { + val exprKey = AttributeReference("k", IntegerType)() + val partitionKeys = Seq(InternalRow(1), InternalRow(2), InternalRow(1)) + val ordering = Seq(SortOrder(exprKey, Ascending)) + val leaf = DummyLeafSafeForKWayMerge( + outputPartitioning = KeyedPartitioning(Seq(exprKey), partitionKeys), + outputOrdering = ordering) + val gpe = GroupPartitionsExec(leaf) + + // SortExec: the pruning condition (!isInstanceOf[SortExec]) stops traversal, so the GPE + // inside is not enabled in any alternative. + assert(!EnsureRequirements.tryEnableSortedMerge( + SortExec(ordering, global = false, child = gpe)).exists(anyGpeEnabled)) + // Exchange produces non-KeyedPartitioning output so the hasKeyedPartitioning half of the + // pruning condition stops traversal; GPE inside is not enabled. + assert(!EnsureRequirements.tryEnableSortedMerge(DummyExchangeExec(gpe)).exists(anyGpeEnabled)) + // Plain unary wrapper above a SortExec: traversal reaches the wrapper but stops at the + // SortExec; GPE inside is still not enabled. + assert(!EnsureRequirements.tryEnableSortedMerge( + DummyPassthroughUnaryExec(SortExec(ordering, global = false, child = gpe))) + .exists(anyGpeEnabled)) + } + } + + private def anyGpeEnabled(plan: SparkPlan): Boolean = + plan.collectFirst { case gpe: GroupPartitionsExec if gpe.enableSortedMerge => true }.isDefined +} + +private case class DummyLeafSafeForKWayMerge( + override val outputOrdering: Seq[SortOrder] = Nil, + override val outputPartitioning: Partitioning = UnknownPartitioning(0) + ) extends LeafExecNode with SafeForKWayMerge { + override protected def doExecute(): RDD[InternalRow] = null + override def output: Seq[Attribute] = Seq.empty +} + +private case class DummyPassthroughUnaryExec(child: SparkPlan) extends UnaryExecNode { + override def output: Seq[Attribute] = child.output + override def outputOrdering: Seq[SortOrder] = child.outputOrdering + override def outputPartitioning: Partitioning = child.outputPartitioning + override protected def doExecute(): RDD[InternalRow] = null + override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = + copy(child = newChild) +} + +// Models a binary join whose output ordering comes from the left child (e.g. SHJ stream=left). +private case class DummyOrderFromLeftBinaryExec(left: SparkPlan, right: SparkPlan) + extends BinaryExecNode { + override def output: Seq[Attribute] = left.output ++ right.output + override def outputOrdering: Seq[SortOrder] = left.outputOrdering + override def outputPartitioning: Partitioning = left.outputPartitioning + override protected def doExecute(): RDD[InternalRow] = null + override protected def withNewChildrenInternal( + newLeft: SparkPlan, newRight: SparkPlan): SparkPlan = + copy(left = newLeft, right = newRight) +} + +// Models a binary join whose outputPartitioning is a PartitioningCollection containing both +// children's partitionings (e.g. SHJ InnerLike), while outputOrdering still comes from the left +// child only. +private case class DummyBothKPBinaryExec(left: SparkPlan, right: SparkPlan) + extends BinaryExecNode { + override def output: Seq[Attribute] = left.output ++ right.output + override def outputOrdering: Seq[SortOrder] = left.outputOrdering + override def outputPartitioning: Partitioning = + PartitioningCollection(Seq(left.outputPartitioning, right.outputPartitioning)) + override protected def doExecute(): RDD[InternalRow] = null + override protected def withNewChildrenInternal( + newLeft: SparkPlan, newRight: SparkPlan): SparkPlan = + copy(left = newLeft, right = newRight) +} + +// Exchange produces non-KeyedPartitioning output (UnknownPartitioning by default); +// do not override outputPartitioning or outputOrdering here. +private case class DummyExchangeExec(child: SparkPlan) extends Exchange { + override def output: Seq[Attribute] = child.output + override protected def doExecute(): RDD[InternalRow] = null + override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = + copy(child = newChild) } From 9bae5a1ecf27d6a7228fdc230aec6aa8d07fd171 Mon Sep 17 00:00:00 2001 From: Yuming Wang Date: Tue, 28 Apr 2026 13:27:27 -0700 Subject: [PATCH 009/286] [SPARK-56587][SQL] Show table names for V2 write nodes in UI ### What changes were proposed in this pull request? This PR modifies the physical execution nodes for DataSourceV2 write operations (such as `AppendDataExec`, `OverwriteByExpressionExec`, `OverwritePartitionsDynamicExec`, `ReplaceDataExec`, and `WriteDeltaExec`) to accept and store the destination `tableName`. It then updates the `nodeName` property in the base `V2ExistingTableWriteExec` trait to include this table name in its output. ### Why are the changes needed? To improve observability and debuggability in the Spark SQL UI and Explain plans. Previously, V2 write nodes were displayed generically (e.g., `AppendData`). With this change, the UI will explicitly show the context of the write operation (e.g., `AppendData catalog.namespace.table_name`), making it much easier for users to understand which tables are being modified. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test and manual test: image ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55510 from wangyum/SPARK-56587. Authored-by: Yuming Wang Signed-off-by: Dongjoon Hyun --- .../datasources/v2/DataSourceV2Strategy.scala | 12 +++++----- .../v2/WriteToDataSourceV2Exec.scala | 18 ++++++++++----- .../connector/DataSourceV2OptionSuite.scala | 16 +++++++------- .../sql/connector/DataSourceV2SQLSuite.scala | 22 +++++++++++++++++++ 4 files changed, 50 insertions(+), 18 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala index d677ff1c4be2b..e03928867e24d 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala @@ -423,7 +423,7 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat } case AppendData(r: DataSourceV2Relation, query, _, _, _, Some(write), _) => - AppendDataExec(planLater(query), refreshCache(r), write) :: Nil + AppendDataExec(planLater(query), refreshCache(r), write, r.name) :: Nil case OverwriteByExpression(r @ ExtractV2Table(v1: SupportsWrite), _, _, _, _, _, Some(write), analyzedQuery) if v1.supports(TableCapability.V1_BATCH_WRITE) => @@ -438,10 +438,10 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat case OverwriteByExpression( r: DataSourceV2Relation, _, query, _, _, _, Some(write), _) => - OverwriteByExpressionExec(planLater(query), refreshCache(r), write) :: Nil + OverwriteByExpressionExec(planLater(query), refreshCache(r), write, r.name) :: Nil case OverwritePartitionsDynamic(r: DataSourceV2Relation, query, _, _, _, Some(write)) => - OverwritePartitionsDynamicExec(planLater(query), refreshCache(r), write) :: Nil + OverwritePartitionsDynamicExec(planLater(query), refreshCache(r), write, r.name) :: Nil case DeleteFromTableWithFilters(r: DataSourceV2Relation, filters) => DeleteFromTableExec(r.table.asDeletable, filters.toArray, refreshCache(r)) :: Nil @@ -487,7 +487,8 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat refreshCache(r), // use the original relation to refresh the cache projections, write, - rd.operation.command) :: Nil + rd.operation.command, + r.name) :: Nil case wd @ WriteDelta(_: DataSourceV2Relation, _, query, r: DataSourceV2Relation, projections, Some(write)) => @@ -496,7 +497,8 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat refreshCache(r), // use the original relation to refresh the cache projections, write, - wd.operation.command) :: Nil + wd.operation.command, + r.name) :: Nil case MergeRows(isSourceRowPresent, isTargetRowPresent, matchedInstructions, notMatchedInstructions, notMatchedBySourceInstructions, checkCardinality, output, child) => diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala index 6bb1eb6f4b6d6..2071024c5b7e5 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala @@ -272,7 +272,8 @@ case class AtomicReplaceTableAsSelectExec( case class AppendDataExec( query: SparkPlan, refreshCache: () => Unit, - write: Write) extends V2ExistingTableWriteExec { + write: Write, + tableName: String) extends V2ExistingTableWriteExec { override protected def withNewChildInternal(newChild: SparkPlan): AppendDataExec = copy(query = newChild) } @@ -290,7 +291,8 @@ case class AppendDataExec( case class OverwriteByExpressionExec( query: SparkPlan, refreshCache: () => Unit, - write: Write) extends V2ExistingTableWriteExec { + write: Write, + tableName: String) extends V2ExistingTableWriteExec { override protected def withNewChildInternal(newChild: SparkPlan): OverwriteByExpressionExec = copy(query = newChild) } @@ -307,7 +309,8 @@ case class OverwriteByExpressionExec( case class OverwritePartitionsDynamicExec( query: SparkPlan, refreshCache: () => Unit, - write: Write) extends V2ExistingTableWriteExec { + write: Write, + tableName: String) extends V2ExistingTableWriteExec { override protected def withNewChildInternal(newChild: SparkPlan): OverwritePartitionsDynamicExec = copy(query = newChild) } @@ -320,7 +323,8 @@ case class ReplaceDataExec( refreshCache: () => Unit, projections: ReplaceDataProjections, write: Write, - rowLevelCommand: RowLevelOperation.Command) extends RowLevelWriteExec { + rowLevelCommand: RowLevelOperation.Command, + tableName: String) extends RowLevelWriteExec { override def writingTask: WritingSparkTask[_] = { projections.metadataProjection match { @@ -364,7 +368,8 @@ case class WriteDeltaExec( refreshCache: () => Unit, projections: WriteDeltaProjections, write: DeltaWrite, - rowLevelCommand: RowLevelOperation.Command) extends RowLevelWriteExec { + rowLevelCommand: RowLevelOperation.Command, + tableName: String) extends RowLevelWriteExec { override lazy val writingTask: WritingSparkTask[_] = { if (projections.metadataProjection.isDefined) { @@ -404,9 +409,12 @@ case class WriteToDataSourceV2Exec( trait V2ExistingTableWriteExec extends V2TableWriteExec { def refreshCache: () => Unit def write: Write + def tableName: String override def stringArgs: Iterator[Any] = Iterator(query, write) + override def nodeName: String = s"${super.nodeName} $tableName" + override val customMetrics: Map[String, SQLMetric] = write.supportedCustomMetrics().map { customMetric => customMetric.name() -> SQLMetrics.createV2CustomMetric(sparkContext, customMetric) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala index 30890200df79d..fbcfdfb20c6ec 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala @@ -109,7 +109,7 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase { collected = df.queryExecution.executedPlan.collect { case CommandResultExec( - _, AppendDataExec(_, _, write), + _, AppendDataExec(_, _, write, _), _) => val append = write.toBatch.asInstanceOf[InMemoryBaseTable#Append] assert(append.info.options.get("write.split-size") === "10") @@ -141,7 +141,7 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase { assert (collected.size == 1) collected = qe.executedPlan.collect { - case AppendDataExec(_, _, write) => + case AppendDataExec(_, _, write, _) => val append = write.toBatch.asInstanceOf[InMemoryBaseTable#Append] assert(append.info.options.get("write.split-size") === "10") } @@ -168,7 +168,7 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase { assert (collected.size == 1) collected = qe.executedPlan.collect { - case AppendDataExec(_, _, write) => + case AppendDataExec(_, _, write, _) => val append = write.toBatch.asInstanceOf[InMemoryBaseTable#Append] assert(append.info.options.get("write.split-size") === "10") } @@ -194,7 +194,7 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase { collected = df.queryExecution.executedPlan.collect { case CommandResultExec( - _, OverwriteByExpressionExec(_, _, write), + _, OverwriteByExpressionExec(_, _, write, _), _) => val append = write.toBatch.asInstanceOf[InMemoryBaseTable#TruncateAndAppend] assert(append.info.options.get("write.split-size") === "10") @@ -227,7 +227,7 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase { assert (collected.size == 1) collected = qe.executedPlan.collect { - case OverwritePartitionsDynamicExec(_, _, write) => + case OverwritePartitionsDynamicExec(_, _, write, _) => val dynOverwrite = write.toBatch.asInstanceOf[InMemoryBaseTable#DynamicOverwrite] assert(dynOverwrite.info.options.get("write.split-size") === "10") } @@ -254,7 +254,7 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase { collected = df.queryExecution.executedPlan.collect { case CommandResultExec( - _, OverwriteByExpressionExec(_, _, write), + _, OverwriteByExpressionExec(_, _, write, _), _) => val append = write.toBatch.asInstanceOf[InMemoryBaseTable#TruncateAndAppend] assert(append.info.options.get("write.split-size") === "10") @@ -287,7 +287,7 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase { assert (collected.size == 1) collected = qe.executedPlan.collect { - case OverwriteByExpressionExec(_, _, write) => + case OverwriteByExpressionExec(_, _, write, _) => val append = write.toBatch.asInstanceOf[InMemoryBaseTable#TruncateAndAppend] assert(append.info.options.get("write.split-size") === "10") } @@ -317,7 +317,7 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase { assert (collected.size == 1) collected = qe.executedPlan.collect { - case OverwriteByExpressionExec(_, _, write) => + case OverwriteByExpressionExec(_, _, write, _) => val append = write.toBatch.asInstanceOf[InMemoryBaseTable#TruncateAndAppend] assert(append.info.options.get("write.split-size") === "10") } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala index d1dc9c282829f..c8f1e2f9ddc23 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala @@ -4302,6 +4302,28 @@ class DataSourceV2SQLSuiteV1Filter } } + test("SPARK-56587: Show table names for V2 write nodes in UI") { + val t1 = s"testcat.ns1.ns2.table_name" + withTable(t1) { + sql(s"CREATE TABLE $t1 (id bigint, data string) USING foo") + val df1 = sql(s"INSERT INTO $t1 VALUES (1, 'a')") + val executed1 = df1.queryExecution.executedPlan + assert(executed1.collect { + case org.apache.spark.sql.execution.CommandResultExec( + _, w: org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec, _) => w + case w: org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec => w + }.head.nodeName.contains("testcat.ns1.ns2.table_name")) + + val df2 = sql(s"INSERT OVERWRITE $t1 VALUES (2, 'b')") + val executed2 = df2.queryExecution.executedPlan + assert(executed2.collect { + case org.apache.spark.sql.execution.CommandResultExec( + _, w: org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec, _) => w + case w: org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec => w + }.head.nodeName.contains("testcat.ns1.ns2.table_name")) + } + } + private def testNotSupportedV2Command( sqlCommand: String, sqlParams: String, From 83af083d6a8d29ad2711102cb4e073690226b88f Mon Sep 17 00:00:00 2001 From: Sandro Sp Date: Tue, 28 Apr 2026 16:29:09 -0700 Subject: [PATCH 010/286] [SPARK-55952][SPARK-55953][SQL] Add ResolveChangelogTable analyzer rule for batch CDC post-processing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? Re-apply of #55508 (commit 881957a, which was reverted in fe6051a) plus a fix for the `ProtoToParsedPlanTestSuite` failures that triggered the revert. The original commit introduces the `ResolveChangelogTable` analyzer rule that post-processes a resolved `DataSourceV2Relation(ChangelogTable)` to inject carry-over removal and/or update detection, fused into a single pass over a `(rowId, _commit_version)`-partitioned Window. To prevent silent wrong results, it also includes an explicit rejection path for streaming CDC reads that would require post-processing. Included changes (re-applied from #55508): - `ResolveChangelogTable` analyzer rule: - **Batch**: applies the requested post-processing transformations. Carry-over removal is a `Filter` on the Window (drops CoW pairs where `min(rowVersion) == max(rowVersion)`). Update detection is a `CASE WHEN` over delete/insert counts (relabels pairs as `update_preimage` / `update_postimage`). The two passes are fused into a single Window. - **Streaming**: throws `INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED` when the requested options would need post-processing. Streams that don't need post-processing pass through unchanged. - **Net changes**: throws `INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED` for both batch and streaming. - Option validation: throws `INVALID_CDC_OPTION.UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL` when `computeUpdates = true` is combined with a carry-over-surfacing connector and `deduplicationMode = none`. - Runtime guard: `INVALID_CDC_OPTION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION` when the connector emits more than one delete or insert for the same `(rowId, _commit_version)` partition. - `Analyzer`: register the rule after `ResolveRelations`. - `InMemoryChangelogCatalog`: `ChangelogProperties` extension so tests can configure post-processing scenarios without a real connector. Additional changes in this PR that were missing from the original commit (the cause of the revert): - `PlanGenerationTestSuite`: switch the `read changes with options` test from `deduplicationMode = netChanges` to `dropCarryovers` since `netChanges` is now rejected up-front by the new rule (`NET_CHANGES_NOT_YET_SUPPORTED`). - Regenerate the corresponding `read_changes_with_options.{json,proto.bin}` query inputs. - Regenerate `streaming_changes_API_with_options.explain` golden file to include the new `resolved: Boolean` field added to `ChangelogTable`. ### Why are the changes needed? Currently, `CHANGES FROM VERSION ... WITH (deduplicationMode = ..., computeUpdates = ...)` parses the options, but they are silently ignored — connector output is returned raw. This PR wires the options to their actual semantics for batch reads, and prevents silent wrong results for streaming reads. ### Does this PR introduce _any_ user-facing change? Yes, for CDC queries against a `Changelog` connector. See the original PR #55508 description for a before/after example. ### How was this patch tested? - `ResolveChangelogTablePostProcessingSuite` exercises the batch rule end-to-end via SQL against `InMemoryChangelogCatalog` (carry-over removal, update detection, their interaction across the option and connector-flag matrix, data-column handling with mixed types, and plan-shape invariants). - `ChangelogResolutionSuite` adds streaming-rejection cases for the two capability flags that would require post-processing. - `ProtoToParsedPlanTestSuite` (724/724) — the suite that previously failed and led to the revert. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 Closes #55588 from gengliangwang/SPARK-55952-resolve-changelog-table-reapply. Lead-authored-by: Sandro Sp Co-authored-by: Sandro Sp Signed-off-by: Gengliang Wang --- .../resources/error/error-conditions.json | 28 + .../sql/connector/catalog/Changelog.java | 9 + .../sql/catalyst/analysis/Analyzer.scala | 1 + .../analysis/ResolveChangelogTable.scala | 312 +++++ .../sql/errors/QueryCompilationErrors.scala | 19 + .../datasources/v2/ChangelogTable.scala | 3 +- .../catalog/InMemoryChangelogCatalog.scala | 71 +- .../spark/sql/PlanGenerationTestSuite.scala | 2 +- ...streaming_changes_API_with_options.explain | 2 +- .../queries/read_changes_with_options.json | 2 +- .../read_changes_with_options.proto.bin | Bin 106 -> 110 bytes .../connector/ChangelogEndToEndSuite.scala | 26 +- .../connector/ChangelogResolutionSuite.scala | 76 +- ...lveChangelogTablePostProcessingSuite.scala | 1034 +++++++++++++++++ 14 files changed, 1552 insertions(+), 33 deletions(-) create mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json index e6c786640e090..ff34214e2ad95 100644 --- a/common/utils/src/main/resources/error/error-conditions.json +++ b/common/utils/src/main/resources/error/error-conditions.json @@ -661,6 +661,19 @@ ], "sqlState" : "42P08" }, + "CHANGELOG_CONTRACT_VIOLATION" : { + "message" : [ + "The Change Data Capture (CDC) connector violated the `Changelog` contract at runtime." + ], + "subClass" : { + "UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION" : { + "message" : [ + "Connector emitted multiple delete or insert rows for the same `(rowId, _commit_version)` partition. The `Changelog` contract requires at most one logical change per row identity per commit when `containsIntermediateChanges() = false`. Either fix the connector to deduplicate intermediate states, or set `containsIntermediateChanges() = true` and use `deduplicationMode = netChanges`." + ] + } + }, + "sqlState" : "XX000" + }, "CHECKPOINT_FILE_CHECKSUM_VERIFICATION_FAILED" : { "message" : [ "Checksum verification failed, the file may be corrupted. File: ", @@ -3278,6 +3291,21 @@ "message" : [ "`startingVersion` is required when `endingVersion` is specified for CDC queries." ] + }, + "NET_CHANGES_NOT_YET_SUPPORTED" : { + "message" : [ + "The `deduplicationMode = netChanges` option on connector `` is not yet supported. Use `deduplicationMode = dropCarryovers` (default) or `deduplicationMode = none` instead." + ] + }, + "STREAMING_POST_PROCESSING_NOT_SUPPORTED" : { + "message" : [ + "Change Data Capture (CDC) streaming reads on connector `` do not yet support post-processing (carry-over removal, update detection, or net change computation). The requested combination of options would require post-processing, which is currently only available for batch reads. Use a batch read, or set `deduplicationMode = none` and `computeUpdates = false` to receive raw change rows in streaming." + ] + }, + "UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL" : { + "message" : [ + "`computeUpdates` cannot be used with `deduplicationMode=none` on connector `` because the connector emits copy-on-write carry-over pairs (`containsCarryoverRows()` returns true) that would be silently mislabeled as updates. Set `deduplicationMode` to `dropCarryovers` or `netChanges`." + ] } }, "sqlState" : "42K03" diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java index 0a811aa0ae4d7..5f2203aa1c379 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java @@ -43,6 +43,15 @@ @Evolving public interface Changelog { + /** Constant for the {@code _change_type} value of a row inserted into the table. */ + String CHANGE_TYPE_INSERT = "insert"; + /** Constant for the {@code _change_type} value of a row deleted from the table. */ + String CHANGE_TYPE_DELETE = "delete"; + /** Constant for the {@code _change_type} value of an update's pre-image row. */ + String CHANGE_TYPE_UPDATE_PREIMAGE = "update_preimage"; + /** Constant for the {@code _change_type} value of an update's post-image row. */ + String CHANGE_TYPE_UPDATE_POSTIMAGE = "update_postimage"; + /** A name to identify this changelog. */ String name(); diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala index a72824f953c08..323a7db9c7ad7 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala @@ -466,6 +466,7 @@ class Analyzer( new ResolveCatalogs(catalogManager) :: ResolveInsertInto :: ResolveRelations :: + ResolveChangelogTable :: ResolvePartitionSpec :: ResolveFieldNameAndPosition :: AddMetadataColumns :: diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala new file mode 100644 index 0000000000000..bdf9b9fed09cc --- /dev/null +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala @@ -0,0 +1,312 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.catalyst.analysis + +import org.apache.spark.sql.catalyst.expressions._ +import org.apache.spark.sql.catalyst.expressions.aggregate.{Count, Max, Min} +import org.apache.spark.sql.catalyst.plans.logical._ +import org.apache.spark.sql.catalyst.rules.Rule +import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 +import org.apache.spark.sql.connector.catalog.{Changelog, ChangelogInfo} +import org.apache.spark.sql.errors.QueryCompilationErrors +import org.apache.spark.sql.execution.datasources.v2.{ChangelogTable, DataSourceV2Relation} +import org.apache.spark.sql.types.{IntegerType, StringType} + +/** + * Post-processes a resolved [[ChangelogTable]] read to apply CDC option semantics + * (carry-over removal, update detection) and to enforce supported option combinations. + * + * Fires after [[ResolveRelations]] has wrapped the connector's [[Changelog]] in a + * [[ChangelogTable]]. Both batch ([[DataSourceV2Relation]]) and streaming + * ([[StreamingRelationV2]]) reads are handled: + * - Batch: the requested post-processing passes are injected as logical operators on top + * of the relation. Carry-over removal and update detection are fused into a single + * pass over a (rowId, _commit_version)-partitioned Window: the Filter drops CoW + * carry-over pairs (same rowVersion on both sides) and the subsequent Project relabels + * real delete+insert pairs as update_preimage / update_postimage. + * - Streaming: post-processing is not yet supported. If the requested options would + * require any post-processing, the rule throws an explicit [[AnalysisException]] to + * prevent silent wrong results. Streams that don't require post-processing pass + * through unchanged. + * + * Net change computation (`deduplicationMode = netChanges`) is not yet implemented and + * is rejected up-front for both batch and streaming. + */ +object ResolveChangelogTable extends Rule[LogicalPlan] { + + /** + * Reserved (`__spark_cdc_*`) column names used internally by post-processing; + * connectors must not emit columns with these names. + */ + object HelperColumn { + final val DelCnt = "__spark_cdc_del_cnt" + final val InsCnt = "__spark_cdc_ins_cnt" + final val MinRv = "__spark_cdc_min_rv" + final val MaxRv = "__spark_cdc_max_rv" + final val RvCnt = "__spark_cdc_rv_cnt" + + val all: Set[String] = Set(DelCnt, InsCnt, MinRv, MaxRv, RvCnt) + } + + override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp { + case rel @ DataSourceV2Relation(table: ChangelogTable, _, _, _, _, _) if !table.resolved => + val changelog = table.changelog + val req = evaluateRequirements(changelog, table.changelogInfo) + + val resolvedRel = rel.copy(table = table.copy(resolved = true)) + var updatedRel: LogicalPlan = resolvedRel + if (req.requiresCarryOverRemoval || req.requiresUpdateDetection) { + updatedRel = addRowLevelPostProcessing( + resolvedRel, changelog, req.requiresCarryOverRemoval, req.requiresUpdateDetection) + } + if (req.requiresNetChanges) { + updatedRel = injectNetChangeComputation(updatedRel, changelog) + } + updatedRel + + case rel @ StreamingRelationV2(_, _, table: ChangelogTable, _, _, _, _, _, _) + if !table.resolved => + // Streaming CDC reads do not yet apply post-processing. Run the same option / + // capability validation as the batch path so silent wrong results are impossible: + // either no post-processing would be required (fall through, return raw stream), + // or we throw an explicit AnalysisException. + val changelog = table.changelog + val req = evaluateRequirements(changelog, table.changelogInfo) + if (req.needsAny) { + throw QueryCompilationErrors.cdcStreamingPostProcessingNotSupported(changelog.name()) + } + rel.copy(table = table.copy(resolved = true)) + } + + // --------------------------------------------------------------------------- + // Option validation & Requirement Computation + // --------------------------------------------------------------------------- + + /** + * Captures which post-processing passes a CDC query requires, derived from the + * user-provided [[ChangelogInfo]] options and the connector-declared [[Changelog]] + * capability flags. + */ + private case class PostProcessingRequirements( + requiresCarryOverRemoval: Boolean, + requiresUpdateDetection: Boolean, + requiresNetChanges: Boolean) { + def needsAny: Boolean = + requiresCarryOverRemoval || requiresUpdateDetection || requiresNetChanges + } + + /** + * Validates CDC option/capability combinations and computes which post-processing + * passes are required. Throws an [[org.apache.spark.sql.AnalysisException]] for + * unsupported or contradictory combinations (currently: `netChanges` deduplication, + * and `computeUpdates` with surfaced carry-overs but no carry-over removal). + */ + private def evaluateRequirements( + changelog: Changelog, + options: ChangelogInfo): PostProcessingRequirements = { + // Net change computation is not yet implemented. + if (options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NET_CHANGES) { + throw QueryCompilationErrors.cdcNetChangesNotYetSupported(changelog.name()) + } + + val requiresCarryOverRemoval = + options.deduplicationMode() != ChangelogInfo.DeduplicationMode.NONE && + changelog.containsCarryoverRows() + val requiresUpdateDetection = + options.computeUpdates() && changelog.representsUpdateAsDeleteAndInsert() + val requiresNetChanges = + options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NET_CHANGES && + changelog.containsIntermediateChanges() + + // If carry-overs are surfaced and update detection is enabled without carry-over + // removal, carry-overs would be falsely classified as updates, leading to wrong + // results. Hence we throw. + if (requiresUpdateDetection && + changelog.containsCarryoverRows() && + options.deduplicationMode() == ChangelogInfo.DeduplicationMode.NONE) { + throw QueryCompilationErrors.cdcUpdateDetectionRequiresCarryOverRemoval( + changelog.name()) + } + + PostProcessingRequirements( + requiresCarryOverRemoval, requiresUpdateDetection, requiresNetChanges) + } + + // --------------------------------------------------------------------------- + // Row Level Post Processing (Update Detection & Carry-over Removal) + // --------------------------------------------------------------------------- + + /** + * Adds row-level post-processing (carry-over removal and/or update detection) on top of + * the given plan. `counts` = per-partition delete and insert change_type row counts over + * `(rowId, _commit_version)`. `rv bounds` = per-partition min/max of `rowVersion`. + * Equal bounds signal a copy-on-write carry-over. + * - both active -> Window(counts + rv bounds) -> Filter -> Project(relabel) -> Drop helpers + * - carry-over only -> Window(counts + rv bounds) -> Filter -> Drop helpers + * - update only -> Window(counts only) -> Project(relabel) -> Drop helpers + * - neither -> not invoked (caller guards this case) + */ + private def addRowLevelPostProcessing( + plan: LogicalPlan, + cl: Changelog, + requiresCarryOverRemoval: Boolean, + requiresUpdateDetection: Boolean): LogicalPlan = { + // Row-version bounds in the Window are needed iff we filter carry-over pairs. + var modifiedPlan = addPostProcessingWindow(plan, cl, + includeRowVersionBounds = requiresCarryOverRemoval) + if (requiresCarryOverRemoval) modifiedPlan = addCarryOverPairFilter(modifiedPlan) + if (requiresUpdateDetection) modifiedPlan = addUpdateRelabelProjection(modifiedPlan) + removeHelperColumns(modifiedPlan) + } + + /** + * Adds a Window node partitioned by (rowId, _commit_version) that computes + * `_del_cnt` and `_ins_cnt` per partition, and, when `includeRowVersionBounds` + * is true, additionally `_min_rv` / `_max_rv` / `_rv_cnt` (min, max and non-null + * count of `Changelog.rowVersion()`). + * + * `_del_cnt` / `_ins_cnt` drive update detection (1 each -> relabel as + * update_preimage / update_postimage). `_min_rv` / `_max_rv` / `_rv_cnt` drive + * carry-over detection (within a delete+insert pair, `_rv_cnt = 2` AND equal + * bounds signal a CoW carry-over). + */ + private def addPostProcessingWindow( + plan: LogicalPlan, + cl: Changelog, + includeRowVersionBounds: Boolean): LogicalPlan = { + val changeTypeAttr = getAttribute(plan, "_change_type") + val rowIdExprs = V2ExpressionUtils.resolveRefs[NamedExpression](cl.rowId().toSeq, plan) + val commitVersionAttr = getAttribute(plan, "_commit_version") + val partitionByCols = rowIdExprs ++ Seq(commitVersionAttr) + val windowSpec = WindowSpecDefinition(partitionByCols, Nil, UnspecifiedFrame) + + val insertIf = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_INSERT)), + Literal(1), Literal(null, IntegerType)) + val deleteIf = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_DELETE)), + Literal(1), Literal(null, IntegerType)) + + val insCntAlias = Alias(WindowExpression( + Count(Seq(insertIf)).toAggregateExpression(), windowSpec), HelperColumn.InsCnt)() + val delCntAlias = Alias(WindowExpression( + Count(Seq(deleteIf)).toAggregateExpression(), windowSpec), HelperColumn.DelCnt)() + val baseAliases = Seq(delCntAlias, insCntAlias) + val rowVersionAliases = if (includeRowVersionBounds) { + val rowVersionExpr = + V2ExpressionUtils.resolveRef[NamedExpression](cl.rowVersion(), plan) + Seq( + Alias(WindowExpression( + Min(rowVersionExpr).toAggregateExpression(), windowSpec), HelperColumn.MinRv)(), + Alias(WindowExpression( + Max(rowVersionExpr).toAggregateExpression(), windowSpec), HelperColumn.MaxRv)(), + Alias(WindowExpression( + Count(Seq(rowVersionExpr)).toAggregateExpression(), windowSpec), HelperColumn.RvCnt)()) + } else { + Seq.empty + } + Window(baseAliases ++ rowVersionAliases, partitionByCols, Nil, plan) + } + + /** + * Adds a Filter node that drops rows belonging to a CoW carry-over pair. + * A pair is a carry-over iff + * `_del_cnt = 1 AND _ins_cnt = 1 AND _rv_cnt = 2 AND _min_rv = _max_rv`. + * The `_rv_cnt = 2` clause guards against a NULL rowVersion silently matching + * `_min_rv = _max_rv` (Spark's min/max skip NULLs). + */ + private def addCarryOverPairFilter(input: LogicalPlan): LogicalPlan = { + val delCnt = getAttribute(input, HelperColumn.DelCnt) + val insCnt = getAttribute(input, HelperColumn.InsCnt) + val minRv = getAttribute(input, HelperColumn.MinRv) + val maxRv = getAttribute(input, HelperColumn.MaxRv) + val rvCnt = getAttribute(input, HelperColumn.RvCnt) + + val isCarryoverPair = And( + And(EqualTo(delCnt, Literal(1L)), EqualTo(insCnt, Literal(1L))), + And(EqualTo(rvCnt, Literal(2L)), EqualTo(minRv, maxRv))) + Filter(Not(isCarryoverPair), input) + } + + /** + * Adds a Project node that rewrites `_change_type` to `update_preimage` / + * `update_postimage` whenever a delete+insert pair is present in the partition. + * Expects the input to expose `_del_cnt` and `_ins_cnt`. + */ + private def addUpdateRelabelProjection(input: LogicalPlan): LogicalPlan = { + val changeTypeAttr = getAttribute(input, "_change_type") + val delCnt = getAttribute(input, HelperColumn.DelCnt) + val insCnt = getAttribute(input, HelperColumn.InsCnt) + + val isUpdate = And( + EqualTo(delCnt, Literal(1L)), + EqualTo(insCnt, Literal(1L))) + val isInvalid = Or(GreaterThan(delCnt, Literal(1L)), GreaterThan(insCnt, Literal(1L))) + val updateType = If(EqualTo(changeTypeAttr, Literal(Changelog.CHANGE_TYPE_INSERT)), + Literal(Changelog.CHANGE_TYPE_UPDATE_POSTIMAGE), + Literal(Changelog.CHANGE_TYPE_UPDATE_PREIMAGE)) + + val raiseInvalid = RaiseError( + Literal("CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION"), + CreateMap(Nil), + StringType) + val caseExpr = CaseWhen(Seq(isInvalid -> raiseInvalid, isUpdate -> updateType), changeTypeAttr) + + val projectList = input.output.map { attr => + if (attr.name == "_change_type") Alias(caseExpr, "_change_type")() + else attr + } + Project(projectList, input) + } + + // --------------------------------------------------------------------------- + // Net Change Computation + // --------------------------------------------------------------------------- + + /** + * Collapses multiple changes per row identity into the net effect. + * Not yet implemented. + */ + private def injectNetChangeComputation( + plan: LogicalPlan, + cl: Changelog): LogicalPlan = { + plan + } + + // --------------------------------------------------------------------------- + // Helpers + // --------------------------------------------------------------------------- + + /** + * Removes any helper columns (see [[HelperColumn]]) that earlier steps added to the + * plan. Helper columns not present in the input are silently ignored, so this method + * can be applied unconditionally regardless of which post-processing steps ran. + */ + private def removeHelperColumns(input: LogicalPlan): LogicalPlan = { + Project(input.output.filterNot(a => HelperColumn.all.contains(a.name)), input) + } + + /** + * Looks up an attribute by name in a plan's output. Throws a clear error if missing -- + * used for required columns like `_change_type` / `_commit_version` / helper columns + * added by earlier steps; a missing column is always a programming error. + */ + private def getAttribute(plan: LogicalPlan, name: String): Attribute = + plan.output.find(_.name == name).getOrElse( + throw new IllegalStateException( + s"Required column '$name' not found in plan output: " + + plan.output.map(_.name).mkString(", "))) +} diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala index b596d2f95391f..02e9f188e0fa4 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala @@ -3862,6 +3862,25 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat messageParameters = Map("catalogName" -> catalogName)) } + def cdcUpdateDetectionRequiresCarryOverRemoval( + changelogName: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CDC_OPTION.UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL", + messageParameters = Map("changelogName" -> changelogName)) + } + + def cdcNetChangesNotYetSupported(changelogName: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", + messageParameters = Map("changelogName" -> changelogName)) + } + + def cdcStreamingPostProcessingNotSupported(changelogName: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", + messageParameters = Map("changelogName" -> changelogName)) + } + def invalidCdcOptionConflictingRangeTypes(): Throwable = { new AnalysisException( errorClass = "INVALID_CDC_OPTION.CONFLICTING_RANGE_TYPES", diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala index 8521df3db2ff0..bb5a03f64990d 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala @@ -33,7 +33,8 @@ import org.apache.spark.sql.util.CaseInsensitiveStringMap */ case class ChangelogTable( changelog: Changelog, - changelogInfo: ChangelogInfo) extends Table with SupportsRead { + changelogInfo: ChangelogInfo, + resolved: Boolean = false) extends Table with SupportsRead { override def name: String = changelog.name diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala index c47ed2668e3b4..3a37b0a84fa26 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryChangelogCatalog.scala @@ -23,6 +23,7 @@ import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.analysis.NoSuchTableException import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.ChangelogRange.{TimestampRange, UnboundedRange, VersionRange} +import org.apache.spark.sql.connector.expressions.{FieldReference, NamedReference} import org.apache.spark.sql.connector.read._ import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, Offset} import org.apache.spark.sql.types._ @@ -44,6 +45,22 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { private var _lastChangelogInfo: Option[ChangelogInfo] = None def lastChangelogInfo: Option[ChangelogInfo] = _lastChangelogInfo + // Per-table overrides for Changelog properties (carry-over rows, intermediate changes, + // update representation, row identity). Tests can set these to exercise post-processing. + private val changelogProperties: mutable.Map[String, ChangelogProperties] = + mutable.Map.empty + + /** + * Override the [[Changelog]] properties returned for a given table. + * Defaults are: containsCarryoverRows=false, containsIntermediateChanges=false, + * representsUpdateAsDeleteAndInsert=false, no rowId, no rowVersion. + */ + def setChangelogProperties( + ident: Identifier, + properties: ChangelogProperties): Unit = { + changelogProperties(ident.toString) = properties + } + override def loadChangelog( ident: Identifier, changelogInfo: ChangelogInfo): Changelog = { @@ -58,8 +75,9 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { // _commit_version is at index numDataCols + 1 (after _change_type) val commitVersionIdx = numDataCols + 1 val filtered = filterByRange(allRows.toSeq, commitVersionIdx, changelogInfo.range()) + val props = changelogProperties.getOrElse(ident.toString, ChangelogProperties()) new InMemoryChangelog( - table.name + "_changelog", table.columns, filtered) + table.name + "_changelog", table.columns, filtered, props) } /** @@ -109,15 +127,42 @@ class InMemoryChangelogCatalog extends InMemoryCatalog { } } +/** + * Configurable properties for [[InMemoryChangelog]] that test cases can use to exercise + * Spark's post-processing (carry-over removal, update detection, net changes). + * + * @param containsCarryoverRows whether the change stream may contain identical CoW pairs + * @param containsIntermediateChanges whether multiple changes per row may exist + * @param representsUpdateAsDeleteAndInsert whether updates appear as raw delete+insert + * @param rowIdNames optional row identity columns as top-level names (e.g. Seq("id")) + * @param rowIdPaths optional row identity paths for nested struct fields + * (e.g. Seq(Seq("payload", "id"))); takes precedence over rowIdNames + * @param rowVersionName optional row version column (e.g. Some("row_commit_version")); + * must be a per-row version that distinguishes carry-overs from + * real updates. Do NOT pass the commit version, which is constant + * within a partition and would cause every delete+insert pair to + * look like a carry-over + */ +case class ChangelogProperties( + containsCarryoverRows: Boolean = false, + containsIntermediateChanges: Boolean = false, + representsUpdateAsDeleteAndInsert: Boolean = false, + rowIdNames: Seq[String] = Seq.empty, + rowIdPaths: Seq[Seq[String]] = Seq.empty, + rowVersionName: Option[String] = None) + /** * A test [[Changelog]] that returns pre-populated change rows. * - * Reports `containsCarryoverRows = false` so Spark skips carry-over removal. + * Properties (carry-over presence, update representation, row identity) are configurable + * via the [[ChangelogProperties]] parameter so tests can exercise different code paths + * in Spark's post-processing analyzer rule. */ class InMemoryChangelog( tableName: String, dataColumns: Array[Column], - changeRows: Seq[InternalRow]) extends Changelog { + changeRows: Seq[InternalRow], + properties: ChangelogProperties = ChangelogProperties()) extends Changelog { private val cdcColumns: Array[Column] = dataColumns ++ Array( Column.create("_change_type", StringType), @@ -128,11 +173,25 @@ class InMemoryChangelog( override def columns(): Array[Column] = cdcColumns - override def containsCarryoverRows(): Boolean = false + override def containsCarryoverRows(): Boolean = properties.containsCarryoverRows + + override def containsIntermediateChanges(): Boolean = properties.containsIntermediateChanges - override def containsIntermediateChanges(): Boolean = false + override def representsUpdateAsDeleteAndInsert(): Boolean = + properties.representsUpdateAsDeleteAndInsert - override def representsUpdateAsDeleteAndInsert(): Boolean = false + override def rowId(): Array[NamedReference] = { + if (properties.rowIdPaths.nonEmpty) { + properties.rowIdPaths.map(parts => FieldReference(parts): NamedReference).toArray + } else { + properties.rowIdNames.map(name => FieldReference.column(name): NamedReference).toArray + } + } + + override def rowVersion(): NamedReference = properties.rowVersionName match { + case Some(name) => FieldReference.column(name) + case None => super.rowVersion() + } override def newScanBuilder( options: CaseInsensitiveStringMap): ScanBuilder = { diff --git a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala index 0c1123f1a76a0..87c01dc4b35cf 100644 --- a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala +++ b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala @@ -403,7 +403,7 @@ class PlanGenerationTestSuite extends ConnectFunSuite with Logging { test("read changes with options") { session.read .option("startingTimestamp", "2026-01-01") - .option("deduplicationMode", "netChanges") + .option("deduplicationMode", "dropCarryovers") .option("computeUpdates", "true") .changes("myTable") } diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_changes_API_with_options.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_changes_API_with_options.explain index a0316c9dba3a3..9666c4ac76ae8 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_changes_API_with_options.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/streaming_changes_API_with_options.explain @@ -1,2 +1,2 @@ ~SubqueryAlias spark_catalog.tempdb.myStreamingTable -+- ~StreamingRelationV2 spark_catalog.tempdb.myStreamingTable_changelog, ChangelogTable(org.apache.spark.sql.connector.catalog.InMemoryChangelog,ChangelogInfo{range=VersionRange[startingVersion=1, endingVersion=Optional.empty, startingBoundInclusive=true, endingBoundInclusive=true], deduplicationMode=DROP_CARRYOVERS, computeUpdates=false}), [startingVersion=1, deduplicationMode=dropCarryovers], [id#0L, _change_type#0, _commit_version#0L, _commit_timestamp#0], org.apache.spark.sql.connector.catalog.InMemoryChangelogCatalog, tempdb.myStreamingTable, name= ++- ~StreamingRelationV2 spark_catalog.tempdb.myStreamingTable_changelog, ChangelogTable(org.apache.spark.sql.connector.catalog.InMemoryChangelog,ChangelogInfo{range=VersionRange[startingVersion=1, endingVersion=Optional.empty, startingBoundInclusive=true, endingBoundInclusive=true], deduplicationMode=DROP_CARRYOVERS, computeUpdates=false},true), [startingVersion=1, deduplicationMode=dropCarryovers], [id#0L, _change_type#0, _commit_version#0L, _commit_timestamp#0], org.apache.spark.sql.connector.catalog.InMemoryChangelogCatalog, tempdb.myStreamingTable, name= diff --git a/sql/connect/common/src/test/resources/query-tests/queries/read_changes_with_options.json b/sql/connect/common/src/test/resources/query-tests/queries/read_changes_with_options.json index ddc20aada18b8..f24d67a6b1121 100644 --- a/sql/connect/common/src/test/resources/query-tests/queries/read_changes_with_options.json +++ b/sql/connect/common/src/test/resources/query-tests/queries/read_changes_with_options.json @@ -6,7 +6,7 @@ "unparsedIdentifier": "myTable", "options": { "startingTimestamp": "2026-01-01", - "deduplicationMode": "netChanges", + "deduplicationMode": "dropCarryovers", "computeUpdates": "true" } } diff --git a/sql/connect/common/src/test/resources/query-tests/queries/read_changes_with_options.proto.bin b/sql/connect/common/src/test/resources/query-tests/queries/read_changes_with_options.proto.bin index 00ab977b46596d9aab51f8953349c1c7f1c8e9b8..4d5c973813268018175b1534b366648d201aa5c3 100644 GIT binary patch delta 52 zcmd1GW9MQLVEDw8K9OBpS(!^PB{ikAASW|9u_QA;&o@6MRfsR8D8Iltv8bprzbv(A HVxS}dk_Hg+ delta 70 zcmd1HV&`HKVEDw8Jds`7LY_-7B{ikAASW|9u_QA;&o@6MRfsDuwZu6iF)ux}SV)YE YFF8NAptK}4v>+w11SrB%QdF7>0O+z7cK`qY diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala index 006b645193023..9622d23122318 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala @@ -418,27 +418,22 @@ class ChangelogEndToEndSuite extends SharedSparkSession { ChangelogInfo.DeduplicationMode.NONE) } - test("changes() passes deduplicationMode and computeUpdates to catalog") { + test("changes() passes computeUpdates to catalog") { catalog.addChangeRows(ident, Seq( makeChangeRow(1L, "a", "insert", 1L, 1000000L))) // DataFrame API spark.read .option("startingVersion", "1") - .option("deduplicationMode", "netChanges") .option("computeUpdates", "true") .changes(fullTableName) .collect() - val info1 = catalog.lastChangelogInfo.get - assert(info1.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) - assert(info1.computeUpdates() === true) + assert(catalog.lastChangelogInfo.get.computeUpdates() === true) // SQL sql(s"SELECT * FROM $fullTableName CHANGES FROM VERSION 1 " + - "WITH (deduplicationMode = 'netChanges', computeUpdates = 'true')").collect() - val info2 = catalog.lastChangelogInfo.get - assert(info2.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) - assert(info2.computeUpdates() === true) + "WITH (computeUpdates = 'true')").collect() + assert(catalog.lastChangelogInfo.get.computeUpdates() === true) } // ---------- Batch: timestamp range ---------- @@ -589,23 +584,20 @@ class ChangelogEndToEndSuite extends SharedSparkSession { // ---------- Streaming: CDC options ---------- - test("streaming changes() passes deduplicationMode and computeUpdates to catalog") { + test("streaming changes() passes computeUpdates to catalog") { catalog.addChangeRows(ident, Seq( makeChangeRow(1L, "a", "insert", 1L, 1000000L))) // DataFrame API val dfApiStream = spark.readStream .option("startingVersion", "1") - .option("deduplicationMode", "netChanges") .option("computeUpdates", "true") .changes(fullTableName) val q1 = dfApiStream.writeStream .format("memory").queryName("cdc_stream_opts_df").start() try { q1.processAllAvailable() - val info1 = catalog.lastChangelogInfo.get - assert(info1.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) - assert(info1.computeUpdates() === true) + assert(catalog.lastChangelogInfo.get.computeUpdates() === true) } finally { q1.stop() } @@ -613,14 +605,12 @@ class ChangelogEndToEndSuite extends SharedSparkSession { // SQL val sqlStream = sql( s"SELECT * FROM STREAM $fullTableName CHANGES FROM VERSION 1 " + - "WITH (deduplicationMode = 'netChanges', computeUpdates = 'true')") + "WITH (computeUpdates = 'true')") val q2 = sqlStream.writeStream .format("memory").queryName("cdc_stream_opts_sql").start() try { q2.processAllAvailable() - val info2 = catalog.lastChangelogInfo.get - assert(info2.deduplicationMode() === ChangelogInfo.DeduplicationMode.NET_CHANGES) - assert(info2.computeUpdates() === true) + assert(catalog.lastChangelogInfo.get.computeUpdates() === true) } finally { q2.stop() } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala index db6817b0c212c..d403db1e62bf9 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala @@ -17,13 +17,14 @@ package org.apache.spark.sql.connector -import java.util +import java.util.Collections import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 import org.apache.spark.sql.connector.catalog._ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.ChangelogRange +import org.apache.spark.sql.connector.expressions.Transform import org.apache.spark.sql.execution.datasources.v2.{ChangelogTable, DataSourceV2Relation} import org.apache.spark.sql.test.SharedSparkSession import org.apache.spark.sql.types.{LongType, StringType} @@ -63,8 +64,8 @@ class ChangelogResolutionSuite extends SharedSparkSession { Array( Column.create("id", LongType), Column.create("data", StringType)), - Array.empty, - new util.HashMap[String, String]()) + Array.empty[Transform], + Collections.emptyMap[String, String]()) val noCdcCat = spark.sessionState.catalogManager.catalog(noCdcCatalogName).asTableCatalog val ident2 = Identifier.of(Array.empty, "test_table") @@ -76,8 +77,8 @@ class ChangelogResolutionSuite extends SharedSparkSession { Array( Column.create("id", LongType), Column.create("data", StringType)), - Array.empty, - new util.HashMap[String, String]()) + Array.empty[Transform], + Collections.emptyMap[String, String]()) } test("CHANGES clause resolves to DataSourceV2Relation with ChangelogTable") { @@ -203,4 +204,69 @@ class ChangelogResolutionSuite extends SharedSparkSession { assert(range.startingVersion() == "1") assert(range.endingVersion().get() == "5") } + + // =========================================================================== + // Streaming post-processing rejection + // =========================================================================== + // + // Streaming CDC reads bypass the post-processing analyzer rule's transformation + // path. To prevent silent wrong results when the requested options would require + // post-processing, the rule throws an explicit AnalysisException for streaming. + + /** Re-creates the test table with non-nullable columns suitable as rowId / rowVersion. */ + private def recreatePostProcessingTable(): Identifier = { + val cat = spark.sessionState.catalogManager.catalog(cdcCatalogName).asTableCatalog + val ident = Identifier.of(Array.empty, "test_table") + if (cat.tableExists(ident)) cat.dropTable(ident) + cat.createTable( + ident, + Array( + Column.create("id", LongType, false), + Column.create("row_commit_version", LongType, false)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + ident + } + + test("DataStreamReader - changes() with carry-over capability throws") { + val ident = recreatePostProcessingTable() + val cat = spark.sessionState.catalogManager + .catalog(cdcCatalogName) + .asInstanceOf[InMemoryChangelogCatalog] + cat.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + spark.readStream + .changes(s"$cdcCatalogName.test_table") + .queryExecution.analyzed + }, + condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", + parameters = Map("changelogName" -> s"$cdcCatalogName.test_table_changelog")) + } + + test("DataStreamReader - changes() with computeUpdates throws") { + val ident = recreatePostProcessingTable() + val cat = spark.sessionState.catalogManager + .catalog(cdcCatalogName) + .asInstanceOf[InMemoryChangelogCatalog] + cat.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + spark.readStream + .option("computeUpdates", "true") + .option("deduplicationMode", "none") + .changes(s"$cdcCatalogName.test_table") + .queryExecution.analyzed + }, + condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", + parameters = Map("changelogName" -> s"$cdcCatalogName.test_table_changelog")) + } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala new file mode 100644 index 0000000000000..353472a035f91 --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ResolveChangelogTablePostProcessingSuite.scala @@ -0,0 +1,1034 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector + +import java.util.Collections + +import org.scalatest.BeforeAndAfterEach + +import org.apache.spark.SparkRuntimeException +import org.apache.spark.sql.{AnalysisException, QueryTest, Row} +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 +import org.apache.spark.sql.connector.catalog.{ + ChangelogProperties, Column, Identifier, InMemoryChangelogCatalog} +import org.apache.spark.sql.connector.catalog.Changelog.{ + CHANGE_TYPE_DELETE, CHANGE_TYPE_INSERT, CHANGE_TYPE_UPDATE_POSTIMAGE, + CHANGE_TYPE_UPDATE_PREIMAGE} +import org.apache.spark.sql.connector.expressions.Transform +import org.apache.spark.sql.execution.datasources.v2.ChangelogTable +import org.apache.spark.sql.test.SharedSparkSession +import org.apache.spark.sql.types.{ + BinaryType, BooleanType, DoubleType, LongType, StringType, StructField, StructType} +import org.apache.spark.unsafe.types.UTF8String + +/** + * Tests for [[org.apache.spark.sql.catalyst.analysis.ResolveChangelogTable]] using the + * in-memory changelog catalog. These tests don't depend on Delta or any specific connector; + * they directly control what the connector "returns" by populating the in-memory changelog + * with hand-crafted change rows. + * + * Each test sets up [[ChangelogProperties]] on the catalog to enable specific post-processing + * paths (carry-over removal, update detection) and then verifies that Spark's analyzer rule + * correctly transforms the plan and produces the expected output. + */ +class ResolveChangelogTablePostProcessingSuite + extends QueryTest + with SharedSparkSession + with BeforeAndAfterEach { + + private val catalogName = "cdc_test_catalog" + private val testTableName = "events" + + override def beforeAll(): Unit = { + super.beforeAll() + spark.conf.set( + s"spark.sql.catalog.$catalogName", + classOf[InMemoryChangelogCatalog].getName) + } + + override def beforeEach(): Unit = { + super.beforeEach() + val cat = catalog + val ident = Identifier.of(Array.empty, testTableName) + if (cat.tableExists(ident)) cat.dropTable(ident) + cat.clearChangeRows(ident) + cat.setChangelogProperties(ident, ChangelogProperties()) + cat.createTable( + ident, + Array( + Column.create("id", LongType), + Column.create("name", StringType), + Column.create("row_commit_version", LongType, false)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + } + + private def catalog: InMemoryChangelogCatalog = { + spark.sessionState.catalogManager + .catalog(catalogName) + .asInstanceOf[InMemoryChangelogCatalog] + } + + private def ident = Identifier.of(Array.empty, testTableName) + + /** + * Helper to create a change row matching schema + * (id, name, row_commit_version, _change_type, _commit_version, _commit_timestamp). + * + * `rowCommitVersion` follows Delta row-tracking semantics: carry-over pairs (CoW-rewritten + * unchanged rows) share the same value on both sides; real updates carry the OLD value on + * the delete side and the NEW value on the insert side. Defaults to `commitVersion` for + * tests that don't exercise carry-over removal. + */ + private def changeRow( + id: Long, + name: String, + changeType: String, + commitVersion: Long, + rowCommitVersion: Long = -1L, + commitTimestamp: Long = 0L): InternalRow = { + val rcv = if (rowCommitVersion == -1L) commitVersion else rowCommitVersion + InternalRow( + id, + UTF8String.fromString(name), + rcv, + UTF8String.fromString(changeType), + commitVersion, + commitTimestamp) + } + + // =========================================================================== + // Carry-Over Removal + // =========================================================================== + + test("carry-over removal drops identical delete+insert pairs") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // v1: insert Alice and Bob (rcv=1 each) + // v2: real delete Alice (preimage carries old rcv=1); + // carry-over for Bob (CoW, rcv unchanged on both sides) + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // carry-over + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) // carry-over (same rcv) + + checkAnswer( + sql( + s"SELECT id, name, _change_type, _commit_version " + + s"FROM $catalogName.$testTableName CHANGES FROM VERSION 1 TO VERSION 2"), + Seq( + Row(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + Row(2L, "Bob", CHANGE_TYPE_INSERT, 1L), + Row(1L, "Alice", CHANGE_TYPE_DELETE, 2L))) + } + + test("deduplicationMode=none keeps all carry-over rows") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) + + checkAnswer( + sql( + s"SELECT id FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (deduplicationMode = 'none')"), + Seq(Row(1L), Row(2L), Row(2L))) + } + + test("NULL rowVersion on one side is NOT silently dropped as carry-over") { + // Regression for a NULL-safety hole: min/max skip NULLs, so _min_rv = _max_rv alone + // would match a pair with one NULL and one non-null rowVersion. The _rv_cnt = 2 + // clause in the carry-over filter prevents that. + // + // The fixture table here declares `row_commit_version` as nullable so the optimizer + // is not allowed to fold IsNull(non-nullable-col) to false; the NULL is a legitimate + // value the guard must defend against. + val nullableRcvTable = "events_nullable_rcv" + val nullableIdent = Identifier.of(Array.empty, nullableRcvTable) + val cat = catalog + if (cat.tableExists(nullableIdent)) cat.dropTable(nullableIdent) + cat.clearChangeRows(nullableIdent) + cat.createTable( + nullableIdent, + Array( + Column.create("id", LongType), + Column.create("name", StringType), + Column.create("row_commit_version", LongType, true)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + cat.setChangelogProperties(nullableIdent, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + cat.addChangeRows(nullableIdent, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2: one side has NULL rowVersion (buggy connector), the other has a real value. + InternalRow(1L, UTF8String.fromString("Alice"), null, + UTF8String.fromString(CHANGE_TYPE_DELETE), 2L, 0L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 5L))) + + checkAnswer( + sql(s"SELECT id, name, _change_type, _commit_version " + + s"FROM $catalogName.$nullableRcvTable CHANGES FROM VERSION 1 TO VERSION 2"), + Seq( + Row(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + Row(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + Row(1L, "Alice", CHANGE_TYPE_INSERT, 2L))) + } + + // =========================================================================== + // Update Detection + // =========================================================================== + + test("update detection relabels delete+insert with different data as update") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = false, // no carry-overs in this test + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + // v2: Alice -> Robert (delete old, insert new) + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Robert", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version " + + s"FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + + assert(descs.contains("1:Alice:insert"), s"v1 insert. Got: ${descs.mkString(",")}") + assert(descs.contains("1:Alice:update_preimage")) + assert(descs.contains("1:Robert:update_postimage")) + // No raw delete/insert at v2 + assert(!descs.contains("1:Alice:delete")) + assert(!descs.contains("1:Robert:insert")) + } + + test("delete and insert in different versions are NOT labeled as update") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 3L))) + + val rows = sql( + s"SELECT _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 3 " + + s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") + .collect() + + assert(!rows.exists(_.getString(0).contains("update_")), + "Delete and insert in different versions should not be labeled as update") + } + + // =========================================================================== + // Composite rowId: partitioning uses every rowId column + // =========================================================================== + // + // With a composite rowId such as Seq("id", "name"), the (rowId, _commit_version) + // window partition must include BOTH columns. A regression that drops one of the + // rowId columns would either falsely merge two different row identities into one + // partition (silently mislabeling unrelated delete/insert pairs as updates) or + // trip the UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION runtime guard. + + test("update detection with composite rowId keeps different (id, name) tuples raw") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id", "name"), + rowVersionName = Some("row_commit_version"))) + + // delete (1, Alice) and insert (1, Bob) at v2. These are DIFFERENT composite + // rowIds; they must NOT be relabeled as update. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}").toSet + + assert(descs == Set("1:Alice:delete", "1:Bob:insert"), + s"Composite rowId must keep different (id, name) tuples raw. Got: $descs") + } + + test("carry-over removal with composite rowId removes pairs per (id, name) tuple") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id", "name"), + rowVersionName = Some("row_commit_version"))) + + // Two independent carry-over pairs at v2, both with id=1 but different names. + // With correct composite-rowId partitioning, each pair lives in its own + // (id, name, _commit_version) partition, has _del_cnt=1 / _ins_cnt=1 and equal + // _min_rv / _max_rv, and gets dropped. With broken (id-only) partitioning, the + // four rows would collapse into one partition with _del_cnt=2 / _ins_cnt=2 and + // the carry-over filter (which requires =1) would keep them all. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), + changeRow(1L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version " + + s"FROM $catalogName.$testTableName CHANGES FROM VERSION 2 TO VERSION 2") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + assert(rows.isEmpty, + s"Both Alice and Bob carry-over pairs at v2 should be removed. Got: ${descs.mkString(",")}") + } + + // =========================================================================== + // No row identity: post-processing skipped + // =========================================================================== + + test("no capability flags -> post-processing not injected in plan") { + // Default ChangelogProperties has no capability flags set; the rule sees nothing to do. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L))) + + val df = sql( + s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + + val plan = df.queryExecution.analyzed.treeString + assert(!plan.contains("__spark_cdc_del_cnt"), + s"Plan must not contain post-processing window helpers. Plan:\n$plan") + assert(!plan.contains("__spark_cdc_ins_cnt"), + s"Plan must not contain post-processing window helpers. Plan:\n$plan") + } + + test("streaming without post-processing options passes through") { + // Streaming reads with no capability flags on the connector and no + // post-processing options must resolve without the rule throwing. + val df = spark.readStream + .option("startingVersion", "1") + .changes(s"$catalogName.$testTableName") + val analyzed = df.queryExecution.analyzed + val plan = analyzed.treeString + assert(!plan.contains("__spark_cdc_del_cnt"), + s"Streaming plan must not contain post-processing helpers. Plan:\n$plan") + + // Positive assertion: the rule actually fired on the streaming relation. Without this, + // a regression that deletes the streaming arm of `ResolveChangelogTable.apply` would + // also pass the absence-of-helpers check above. + val tableResolved = analyzed.collectFirst { + case rel: StreamingRelationV2 if rel.table.isInstanceOf[ChangelogTable] => + rel.table.asInstanceOf[ChangelogTable].resolved + } + assert(tableResolved.contains(true), + s"Expected ChangelogTable to be marked resolved by the rule. Plan:\n$plan") + } + + test("streaming with post-processing options is rejected") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + exception = intercept[AnalysisException] { + spark.readStream + .option("startingVersion", "1") + .changes(s"$catalogName.$testTableName") + .queryExecution.analyzed + }, + condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", + parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) + } + + // =========================================================================== + // Combined + // =========================================================================== + + test("carry-over removal and update detection combined") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // v1: insert Alice (rcv=1), Bob (rcv=1) + // v2: Alice carry-over (CoW, rcv unchanged), Bob real update (old rcv=1, new rcv=2) + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // carry-over + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), // carry-over + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), // update preimage + changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) // update postimage + + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}").toSet + + // v1 inserts + assert(descs.contains("1:Alice:insert")) + assert(descs.contains("2:Bob:insert")) + // Alice carry-over dropped + assert(!descs.contains("1:Alice:delete")) + // Bob -> Robert as update + assert(descs.contains("2:Bob:update_preimage")) + assert(descs.contains("2:Robert:update_postimage")) + // Should be exactly 4 rows + assert(rows.length == 4, s"Expected 4 rows, got ${rows.length}: ${descs.mkString(",")}") + } + + // =========================================================================== + // computeUpdates default (false) keeps raw delete+insert + // =========================================================================== + + test("without computeUpdates, delete+insert with different data stays raw") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // Alice: carry-over (CoW, rcv unchanged on both sides) + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), + // Bob -> Robert: real change (old rcv on pre, new rcv on post) + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) + + // Default computeUpdates=false: do NOT relabel, but DO drop carry-overs + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + + assert(descs.contains("2:Bob:delete"), s"Bob delete remains raw. Got: ${descs.mkString(",")}") + assert(descs.contains("2:Robert:insert"), "Robert insert remains raw") + assert(!descs.exists(_.contains("update_")), "No update_* without computeUpdates") + assert(!descs.contains("1:Alice:delete"), "Alice carry-over removed") + } + + test("update detection on pure inserts leaves them as inserts") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .collect() + + assert(rows.length == 2) + assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT), + s"Pure inserts must stay 'insert'. Got: ${rows.map(_.getString(1)).mkString(",")}") + } + + // =========================================================================== + // Keep Carry-over Rows and deduplication flag tests + // =========================================================================== + + test("computeUpdates with deduplicationMode=none is rejected on COW connector") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 " + + s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") + }, + condition = "INVALID_CDC_OPTION.UPDATE_DETECTION_REQUIRES_CARRY_OVER_REMOVAL", + parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) + } + + test("computeUpdates with deduplicationMode=none is allowed on non-COW connector") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = false, // MOR-style: no carry-overs possible + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + // v2: Alice -> Robert (delete old, insert new) + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Robert", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 " + + s"WITH (computeUpdates = 'true', deduplicationMode = 'none')") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + assert(descs.contains("1:Alice:update_preimage"), + s"Expected Alice update_preimage. Got: ${descs.mkString(",")}") + assert(descs.contains("1:Robert:update_postimage"), + s"Expected Robert update_postimage. Got: ${descs.mkString(",")}") + } + + // =========================================================================== + // Contract enforcement: at most one delete + one insert per (rowId, version) + // =========================================================================== + // + // With `representsUpdateAsDeleteAndInsert = true` and `containsIntermediateChanges = false`, + // the `Changelog` contract guarantees at most one logical change per (rowId, _commit_version) + // partition. The update-relabel projection enforces this at runtime: if it sees more than one + // delete or more than one insert in a partition, it raises + // INVALID_CDC_OPTION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION instead of silently + // mislabeling extra rows as updates. + + test("update detection raises on multiple inserts for same (rowId, _commit_version)") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // Contract violation: 2 inserts for id=1 at v2. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Alice2", CHANGE_TYPE_INSERT, 2L), + changeRow(1L, "Alice3", CHANGE_TYPE_INSERT, 2L))) + + checkError( + intercept[SparkRuntimeException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") + .collect() + }, + condition = "CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION", + parameters = Map.empty) + } + + test("update detection raises on multiple deletes for same (rowId, _commit_version)") { + catalog.setChangelogProperties(ident, ChangelogProperties( + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // Contract violation: 2 deletes for id=1 at v2. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Alice2", CHANGE_TYPE_DELETE, 2L), + changeRow(1L, "Alice3", CHANGE_TYPE_INSERT, 2L))) + + checkError( + intercept[SparkRuntimeException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (computeUpdates = 'true')") + .collect() + }, + condition = "CHANGELOG_CONTRACT_VIOLATION.UNEXPECTED_MULTIPLE_CHANGES_PER_ROW_VERSION", + parameters = Map.empty) + } + + // =========================================================================== + // Net changes deduplication: not yet supported + // =========================================================================== + // + // `deduplicationMode = netChanges` collapses multiple changes per row identity into the + // net effect. It is not yet implemented in [[ResolveChangelogTable]]. + + test("deduplicationMode=netChanges is rejected when connector emits intermediate changes") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsIntermediateChanges = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 " + + s"WITH (deduplicationMode = 'netChanges')") + }, + condition = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", + parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) + } + + test("deduplicationMode=netChanges is rejected even when connector has no intermediate changes") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsIntermediateChanges = false, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + checkError( + intercept[AnalysisException] { + sql(s"SELECT * FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 " + + s"WITH (deduplicationMode = 'netChanges')") + }, + condition = "INVALID_CDC_OPTION.NET_CHANGES_NOT_YET_SUPPORTED", + parameters = Map("changelogName" -> s"$catalogName.${testTableName}_changelog")) + } + + // =========================================================================== + // Range edge cases + // =========================================================================== + + test("multiple operations across versions") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + // v1: insert 3 rows (rcv=1 each) + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2: delete Alice (preimage carries old rcv=1); CoW carry-overs for Bob/Charlie + // keep rcv=1 on both sides (row unchanged). + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L), + // v3: update Bob -> Robert (old rcv=1, new rcv=3); CoW carry-over for Charlie (rcv=1) + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 3L, rowCommitVersion = 1L), + changeRow(2L, "Robert", CHANGE_TYPE_INSERT, 3L, rowCommitVersion = 3L), + changeRow(3L, "Charlie", CHANGE_TYPE_DELETE, 3L, rowCommitVersion = 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L, rowCommitVersion = 1L), + // v4: insert Diana (rcv=4) + changeRow(4L, "Diana", CHANGE_TYPE_INSERT, 4L, rowCommitVersion = 4L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 4 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}").toSet + + // v1 + assert(descs.contains("1:Alice:insert:v1")) + assert(descs.contains("2:Bob:insert:v1")) + assert(descs.contains("3:Charlie:insert:v1")) + // v2 + assert(descs.contains("1:Alice:delete:v2")) + assert(!descs.contains("2:Bob:delete:v2"), "Bob carry-over dropped") + assert(!descs.contains("3:Charlie:delete:v2"), "Charlie carry-over dropped") + // v3 + assert(descs.contains("2:Bob:update_preimage:v3")) + assert(descs.contains("2:Robert:update_postimage:v3")) + assert(!descs.contains("3:Charlie:delete:v3"), "Charlie carry-over dropped in v3") + // v4 + assert(descs.contains("4:Diana:insert:v4")) + } + + test("larger insert batch returns all rows") { + catalog.addChangeRows(ident, (1 to 5).map(i => + changeRow(i.toLong, ('A' + i - 1).toChar.toString, CHANGE_TYPE_INSERT, 1L))) + + val rows = sql( + s"SELECT id, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 1 WITH (deduplicationMode = 'none')") + .collect() + + assert(rows.length == 5) + assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT)) + } + + test("DELETE all rows: no carry-over inserts at v2") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // v1 inserts carry rcv=1; v2 deletes carry the old rcv=1 (rcv tracks last modification) + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2") + .orderBy("_commit_version", "id") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") + + assert(descs.contains("1:Alice:insert:v1")) + assert(descs.contains("2:Bob:insert:v1")) + assert(descs.contains("1:Alice:delete:v2")) + assert(descs.contains("2:Bob:delete:v2")) + assert(!descs.exists(_.contains("insert:v2")), "No inserts at v2") + } + + test("UPDATE all rows: every row gets update_pre/postimage, no carry-overs") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // Every v2 row is a real update: delete side carries old rcv=1, insert side new rcv=2. + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Alice_updated", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L), + changeRow(2L, "Bob", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(2L, "Bob_updated", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}").toSet + + assert(descs.contains("1:Alice:update_preimage:v2")) + assert(descs.contains("1:Alice_updated:update_postimage:v2")) + assert(descs.contains("2:Bob:update_preimage:v2")) + assert(descs.contains("2:Bob_updated:update_postimage:v2")) + assert(rows.length == 6, s"Expected 2 inserts + 2 pre + 2 post. Got ${rows.length}") + } + + test("append-only workload: all inserts, no carry-over needed") { + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) + + val rows = sql( + s"SELECT id, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 3") + .collect() + + assert(rows.length == 3) + assert(rows.forall(_.getString(1) == CHANGE_TYPE_INSERT)) + } + + test("carry-over removal with many rows: only real change remains") { + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + // 10 inserts at v1 (rcv=1 each). At v2: delete row 5; CoW writes 9 carry-over pairs + // (rcv unchanged since v1, i.e. rcv=1 on both sides) plus 1 real delete (rcv=1, old). + val v1Inserts = (1 to 10).map(i => + changeRow( + i.toLong, ('A' + i - 1).toChar.toString, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L)) + val v2Carryovers = (1 to 10).filter(_ != 5).flatMap { i => + val name = ('A' + i - 1).toChar.toString + Seq( + changeRow(i.toLong, name, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(i.toLong, name, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L)) + } + val v2RealDelete = Seq(changeRow(5L, "E", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L)) + catalog.addChangeRows(ident, v1Inserts ++ v2Carryovers ++ v2RealDelete) + + val rows = sql( + s"SELECT id, name, _change_type FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}") + assert(rows.length == 1, + s"Only 1 real change should remain (9 carry-overs dropped). Got: ${descs.mkString(",")}") + assert(descs.contains("5:E:delete")) + } + + test("carry-over removal with mixed types (DOUBLE, BOOLEAN, BINARY)") { + val mixedTable = "events_mixed" + val mixedIdent = Identifier.of(Array.empty, mixedTable) + val cat = catalog + if (cat.tableExists(mixedIdent)) cat.dropTable(mixedIdent) + cat.clearChangeRows(mixedIdent) + cat.createTable( + mixedIdent, + Array( + Column.create("id", LongType), + Column.create("name", StringType), + Column.create("score", DoubleType), + Column.create("active", BooleanType), + Column.create("payload", BinaryType), + Column.create("row_commit_version", LongType, false)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + cat.setChangelogProperties(mixedIdent, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + def mixedRow( + id: Long, name: String, score: Double, active: Boolean, payload: Array[Byte], + ct: String, v: Long, rowCommitVersion: Long): InternalRow = { + InternalRow( + id, UTF8String.fromString(name), score, active, payload, rowCommitVersion, + UTF8String.fromString(ct), v, 0L) + } + + val alicePayload = Array[Byte](1, 2, 3) + val bobPayload = Array[Byte](4, 5, 6) + + cat.addChangeRows(mixedIdent, Seq( + mixedRow( + 1L, "Alice", 95.5, true, alicePayload, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + mixedRow( + 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2: update Alice's score (old rcv=1, new rcv=2); Bob is carry-over (rcv unchanged) + mixedRow( + 1L, "Alice", 95.5, true, alicePayload, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + mixedRow( + 1L, "Alice", 99.0, true, alicePayload, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L), + mixedRow( + 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + mixedRow( + 2L, "Bob", 87.3, false, bobPayload, CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 1L))) + + val rows = sql( + s"SELECT id, name, score, active, _change_type FROM $catalogName.$mixedTable " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "id", "_change_type") + .collect() + + val descs = rows.map(r => s"${r.getLong(0)}:${r.getString(4)}") + assert(descs.contains("1:update_preimage")) + assert(descs.contains("1:update_postimage")) + assert(!descs.contains("2:delete"), + s"Bob carry-over must be dropped despite DOUBLE/BOOLEAN/BINARY. Got: " + + descs.mkString(",")) + + val pre = rows.find(r => + r.getLong(0) == 1L && r.getString(4) == CHANGE_TYPE_UPDATE_PREIMAGE).get + val post = rows.find(r => + r.getLong(0) == 1L && r.getString(4) == CHANGE_TYPE_UPDATE_POSTIMAGE).get + assert(pre.getDouble(2) == 95.5) + assert(post.getDouble(2) == 99.0) + } + + // =========================================================================== + // Regression: nested rowId + nested rowVersion end-to-end + // =========================================================================== + + // End-to-end check that nested rowId paths (e.g. `payload.id`) are resolved on the plan + // and threaded through carry-over detection. The pair survives the filter because the + // row_commit_version differs across delete/insert, not because of any sibling-field data. + test("nested rowId path resolves correctly through carry-over filter") { + val nestedTable = "events_nested" + val nestedIdent = Identifier.of(Array.empty, nestedTable) + val cat = catalog + if (cat.tableExists(nestedIdent)) cat.dropTable(nestedIdent) + cat.clearChangeRows(nestedIdent) + + val payloadType = StructType(Seq( + StructField("id", LongType), + StructField("value", StringType))) + + cat.createTable( + nestedIdent, + Array( + Column.create("payload", payloadType), + Column.create("row_commit_version", LongType, false)), + Array.empty[Transform], + Collections.emptyMap[String, String]()) + + cat.setChangelogProperties(nestedIdent, ChangelogProperties( + containsCarryoverRows = true, + rowIdPaths = Seq(Seq("payload", "id")), + rowVersionName = Some("row_commit_version"))) + + def nestedRow( + id: Long, value: String, ct: String, v: Long, rowCommitVersion: Long): InternalRow = { + InternalRow( + InternalRow(id, UTF8String.fromString(value)), + rowCommitVersion, + UTF8String.fromString(ct), v, 0L) + } + + cat.addChangeRows(nestedIdent, Seq( + nestedRow(1L, "original", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2 update: rowId same, rowVersion differs (old rcv=1 on preimage, new rcv=2 on postimage) + nestedRow(1L, "original", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + nestedRow(1L, "CHANGED", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) + + val rows = sql( + s"SELECT payload.id AS id, payload.value AS value, _change_type, _commit_version " + + s"FROM $catalogName.$nestedTable CHANGES FROM VERSION 1 TO VERSION 2") + .orderBy("_commit_version", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") + + assert(descs.contains("1:original:insert:v1"), + s"v1 insert must survive. Got: ${descs.mkString(",")}") + assert(descs.contains("1:original:delete:v2"), + s"v2 delete must survive (payload.value differs from insert). Got: ${descs.mkString(",")}") + assert(descs.contains("1:CHANGED:insert:v2"), + s"v2 insert must survive (payload.value differs from delete). Got: ${descs.mkString(",")}") + assert(rows.length == 3, + s"Expected 3 rows (v1 insert + v2 delete + v2 insert). Got ${rows.length}: " + + descs.mkString(",")) + } + + // =========================================================================== + // No-op UPDATE is correctly preserved as update_preimage/postimage + // =========================================================================== + + test("no-op UPDATE is labeled as update (row_commit_version differs on pre/post)") { + // A no-op UPDATE bumps row_commit_version even when data is byte-identical, so the + // delete side carries the OLD rcv and the insert side the NEW rcv. Window post-processing + // sees different rowVersions, treats this as a real change, and labels both rows as + // update_preimage / update_postimage. + catalog.setChangelogProperties(ident, ChangelogProperties( + containsCarryoverRows = true, + representsUpdateAsDeleteAndInsert = true, + rowIdNames = Seq("id"), + rowVersionName = Some("row_commit_version"))) + + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L, rowCommitVersion = 1L), + // v2 no-op update: identical data, but rcv differs (Delta bumps it on any UPDATE) + changeRow(1L, "Alice", CHANGE_TYPE_DELETE, 2L, rowCommitVersion = 1L), + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 2L, rowCommitVersion = 2L))) + + val rows = sql( + s"SELECT id, name, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 TO VERSION 2 WITH (computeUpdates = 'true')") + .orderBy("_commit_version", "_change_type") + .collect() + + val descs = rows.map(r => + s"${r.getLong(0)}:${r.getString(1)}:${r.getString(2)}:v${r.getLong(3)}") + + assert(descs.contains("1:Alice:insert:v1")) + assert(descs.contains("1:Alice:update_preimage:v2"), + s"No-op UPDATE preimage must be labeled. Got: ${descs.mkString(",")}") + assert(descs.contains("1:Alice:update_postimage:v2"), + s"No-op UPDATE postimage must be labeled. Got: ${descs.mkString(",")}") + assert(rows.length == 3, + s"Expected v1 insert + v2 update pre/post = 3 rows. Got ${rows.length}") + } + + // =========================================================================== + // Baseline (range syntax / connector range filtering -- rule bypassed via + // deduplicationMode = 'none'; included as smoke tests for the SQL surface). + // =========================================================================== + + test("baseline: single-version range FROM VERSION X TO VERSION X") { + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 1L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 2L))) + + val rows = sql( + s"SELECT id, _change_type, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 2 TO VERSION 2 WITH (deduplicationMode = 'none')") + .collect() + + assert(rows.length == 1, s"Single version: 1 row. Got ${rows.length}") + assert(rows(0).getLong(0) == 3L) + assert(rows(0).getString(1) == CHANGE_TYPE_INSERT) + } + + test("baseline: EXCLUSIVE start bound skips the start version") { + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) + + val rows = sql( + s"SELECT id, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 EXCLUSIVE TO VERSION 3 " + + s"WITH (deduplicationMode = 'none')") + .orderBy("_commit_version") + .collect() + + assert(!rows.exists(_.getLong(1) == 1L), "v1 must be excluded") + assert(rows.exists(_.getLong(0) == 2L), "Bob (v2) included") + assert(rows.exists(_.getLong(0) == 3L), "Charlie (v3) included") + } + + test("baseline: open-ended range (no TO clause) reads to latest") { + catalog.addChangeRows(ident, Seq( + changeRow(1L, "Alice", CHANGE_TYPE_INSERT, 1L), + changeRow(2L, "Bob", CHANGE_TYPE_INSERT, 2L), + changeRow(3L, "Charlie", CHANGE_TYPE_INSERT, 3L))) + + val rows = sql( + s"SELECT id, _commit_version FROM $catalogName.$testTableName " + + s"CHANGES FROM VERSION 1 WITH (deduplicationMode = 'none')") + .orderBy("_commit_version", "id") + .collect() + + assert(rows.length == 3, s"Open-ended range should see all 3. Got ${rows.length}") + assert(rows.exists(r => r.getLong(0) == 3L && r.getLong(1) == 3L)) + } +} From 18524a039039c64f9ccef7f3734ae05f92b8815b Mon Sep 17 00:00:00 2001 From: chenhao-db Date: Wed, 29 Apr 2026 12:59:34 +0900 Subject: [PATCH 011/286] [SPARK-56134][SQL] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError ### What changes were proposed in this pull request? This is very similar to https://github.com/apache/spark/pull/20779. When a generated code has split classes and they try to access protected fields in `BufferedRowIterator`, an `IllegalAccessError` will happen. I am not making `partitionIndex` public too because I cannot find a real example that can trigger an `IllegalAccessError` on it too. In the code base, `partitionIndex` is mostly accessed via `addPartitionInitializationStatement`. Since it is used in the partition initialization code, not in split classes, there should be no issue with it. ### Why are the changes needed? Fix runtime error in large queries. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #54942 from chenhao-db/SPARK-56134. Authored-by: chenhao-db Signed-off-by: Hyukjin Kwon --- .../sql/execution/BufferedRowIterator.java | 3 ++- .../execution/WholeStageCodegenSuite.scala | 23 +++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java b/sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java index 3d0511b7ba838..52357acf3c7d6 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java @@ -33,7 +33,8 @@ public abstract class BufferedRowIterator { protected LinkedList currentRows = new LinkedList<>(); // used when there is no column in output - protected UnsafeRow unsafeRow = new UnsafeRow(0); + // Keep it public for codegen to access. + public UnsafeRow unsafeRow = new UnsafeRow(0); private long startTimeNs = System.nanoTime(); protected int partitionIndex = -1; diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala index 069bfc72e106e..e5b9e7016841e 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala @@ -777,6 +777,29 @@ class WholeStageCodegenSuite extends SharedSparkSession } } + test("SPARK-56134: Codegen working for empty output") { + // Create a balanced tree of AND conditions. This prevents generating a very deep tree, + // which can cause stack overflow. + def balancedAnd(cols: Seq[String]): String = cols match { + case Seq(single) => single + case seq => + val (left, right) = seq.splitAt(seq.length / 2) + balancedAnd(left) + " and " + balancedAnd(right) + } + + withTempPath { dir => + val path = dir.getCanonicalPath + sql("select array(0) as value from range(0, 1, 1, 1)") + .write.mode(SaveMode.Overwrite).parquet(path) + + val numConditions = 1000 + val conditions = (0 until numConditions).map(i => s"value <= array($i)") + val condition = balancedAnd(conditions) + val df = spark.read.parquet(path).filter(condition).selectExpr() + assert(df.limit(1).selectExpr("count(*)").collect() === Array(Row(1))) + } + } + test("SPARK-25767: Lazy evaluated stream of expressions handled correctly") { val a = Seq(1).toDF("key") val b = Seq((1, "a")).toDF("key", "value") From 7e95673cf044541136e65c1f9c2deb5fc01e1cf3 Mon Sep 17 00:00:00 2001 From: Sandro Sp Date: Tue, 28 Apr 2026 21:35:52 -0700 Subject: [PATCH 012/286] [SPARK-55951][SQL] Add ChangelogTable schema validation and INVALID_CHANGELOG_SCHEMA error class ### What changes were proposed in this pull request? This is **PR 1 of a split** of #55426 (see the [split suggestion](https://github.com/apache/spark/pull/55426#issuecomment-4292375876) for the full plan). Can merge in any order, but 1 (#55507) < 2 (#55508) would be preferable. For more context, see [discussion](https://lists.apache.org/thread/dhxx6pohs7fvqc3knzhtoj4tbcgrwxts) posted to [devspark.apache.org](https://lists.apache.org/list.html?devspark.apache.org) and linked [SPIP](https://docs.google.com/document/d/1-4rCS3vsGIyhwnkAwPsEaqyUDg-AuVkdrYLotFPw0U0/edit?tab=t.0#heading=h.m1700lw4wsoj). Validates the CDC metadata columns and row-identity presence returned by a `Changelog` connector at relation construction time, and introduces a dedicated error class to report the failure at analysis time rather than later at execution time with a less helpful error. - `ChangelogTable.validateSchema`: fail-fast checks that the connector schema contains the required metadata columns (`_change_type` as StringType, `_commit_version` of connector-defined type, `_commit_timestamp` as TimestampType), and that `rowId()` returns a non-empty array when a capability requires row identity. `rowVersion()` is invoked when a capability requires it and surfaces the default `UnsupportedOperationException` directly if the connector has not overridden it. References can be top-level or nested (e.g. Delta's `_metadata.row_commit_version`). Invoked from the `ChangelogTable` constructor. - New error class `INVALID_CHANGELOG_SCHEMA` with sub-classes `MISSING_COLUMN`, `INVALID_COLUMN_TYPE`, `MISSING_ROW_ID`. - Matching `QueryCompilationErrors` helpers for each sub-class. - rowVersion nullability is enforced at runtime in the carry-over filter in #55508 via `count(rowVersion) = 2` (see the [#55426 NULL-safety thread](https://github.com/apache/spark/pull/55426#discussion_r3120231839) for rationale). rowId nullability is not enforced. It is covered by the `Changelog.rowId()` Javadoc contract. ### Why are the changes needed? Gives connector implementors a clear analysis-time error message for misshapen CDC schemas instead of an opaque execution-time failure. Background on the original PR and its [discussion thread](https://lists.apache.org/thread/dhxx6pohs7fvqc3knzhtoj4tbcgrwxts). ### Does this PR introduce _any_ user-facing change? Yes, for connector implementors. A connector that returns an invalid changelog schema (missing or wrong-typed metadata column, or advertising a capability requiring row identity without declaring `rowId()`) now fails at analysis time with `INVALID_CHANGELOG_SCHEMA.*`. A connector that advertises a capability requiring `rowId()` or `rowVersion()` without implementing the method surfaces the default `UnsupportedOperationException` at analysis time. ### How was this patch tested? Added schema-validation cases to `ChangelogResolutionSuite` covering: - Missing metadata column: `_change_type`, `_commit_version`, `_commit_timestamp`. - Wrong data type: `_change_type` non-String, `_commit_timestamp` non-Timestamp. - Connector-defined `_commit_version` type accepted (Integer, Long, String). - Valid schema with data columns passes. - Nested rowId and rowVersion references (Delta-style `_metadata.row_id` / `_metadata.row_commit_version`) pass. - `MISSING_ROW_ID` triggered by `representsUpdateAsDeleteAndInsert = true`. - `MISSING_ROW_ID` triggered by `containsIntermediateChanges = true`. - Default `UnsupportedOperationException` on `rowId()` surfaces when a capability requires it. - Default `UnsupportedOperationException` on `rowVersion()` surfaces when a capability requires it. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 Closes #55507 from SanJSp/SPARK-55668-PR1-changelog-schema-validation. Lead-authored-by: Sandro Sp Co-authored-by: Gengliang Wang Signed-off-by: Gengliang Wang --- .../resources/error/error-conditions.json | 23 ++ .../sql/errors/QueryCompilationErrors.scala | 29 +++ .../datasources/v2/ChangelogTable.scala | 43 ++++ .../connector/ChangelogEndToEndSuite.scala | 137 ++++++------ .../connector/ChangelogResolutionSuite.scala | 203 +++++++++++++++++- 5 files changed, 365 insertions(+), 70 deletions(-) diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json index ff34214e2ad95..87e645ef2b0f0 100644 --- a/common/utils/src/main/resources/error/error-conditions.json +++ b/common/utils/src/main/resources/error/error-conditions.json @@ -3310,6 +3310,29 @@ }, "sqlState" : "42K03" }, + "INVALID_CHANGELOG_SCHEMA" : { + "message" : [ + "The Change Data Capture (CDC) schema returned by connector is invalid." + ], + "subClass" : { + "INVALID_COLUMN_TYPE" : { + "message" : [ + "Column `` has type , expected ." + ] + }, + "MISSING_COLUMN" : { + "message" : [ + "Required column `` is missing." + ] + }, + "MISSING_ROW_ID" : { + "message" : [ + "Connector advertises one or more post-processing properties (`containsCarryoverRows`, `representsUpdateAsDeleteAndInsert`, `containsIntermediateChanges`) that require row identity, but `Changelog.rowId()` returned an empty array." + ] + } + }, + "sqlState" : "42K03" + }, "INVALID_CLONE_SESSION_REQUEST" : { "message" : [ "Invalid session clone request." diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala index 02e9f188e0fa4..f369317b4b0a5 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala @@ -3881,6 +3881,35 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat messageParameters = Map("changelogName" -> changelogName)) } + def changelogMissingColumnError( + changelogName: String, columnName: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CHANGELOG_SCHEMA.MISSING_COLUMN", + messageParameters = Map( + "changelogName" -> changelogName, + "columnName" -> columnName)) + } + + def changelogInvalidColumnTypeError( + changelogName: String, + columnName: String, + expectedType: String, + actualType: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CHANGELOG_SCHEMA.INVALID_COLUMN_TYPE", + messageParameters = Map( + "changelogName" -> changelogName, + "columnName" -> columnName, + "expectedType" -> expectedType, + "actualType" -> actualType)) + } + + def changelogMissingRowIdError(changelogName: String): AnalysisException = { + new AnalysisException( + errorClass = "INVALID_CHANGELOG_SCHEMA.MISSING_ROW_ID", + messageParameters = Map("changelogName" -> changelogName)) + } + def invalidCdcOptionConflictingRangeTypes(): Throwable = { new AnalysisException( errorClass = "INVALID_CDC_OPTION.CONFLICTING_RANGE_TYPES", diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala index bb5a03f64990d..56d79c7c0da37 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ChangelogTable.scala @@ -22,6 +22,8 @@ import java.util.{EnumSet => JEnumSet, Set => JSet} import org.apache.spark.sql.connector.catalog.{Changelog, ChangelogInfo, Column, SupportsRead, Table, TableCapability} import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ, MICRO_BATCH_READ} import org.apache.spark.sql.connector.read.ScanBuilder +import org.apache.spark.sql.errors.QueryCompilationErrors +import org.apache.spark.sql.types.{DataType, StringType, TimestampType} import org.apache.spark.sql.util.CaseInsensitiveStringMap /** @@ -36,6 +38,11 @@ case class ChangelogTable( changelogInfo: ChangelogInfo, resolved: Boolean = false) extends Table with SupportsRead { + // Validate that the connector returned a schema with the required CDC metadata columns + // and correct types. `_commit_version` is connector-defined per the Changelog contract, + // so its type is not checked. + ChangelogTable.validateSchema(changelog) + override def name: String = changelog.name override def columns: Array[Column] = changelog.columns @@ -46,3 +53,39 @@ case class ChangelogTable( override def capabilities: JSet[TableCapability] = JEnumSet.of(BATCH_READ, MICRO_BATCH_READ) } + +object ChangelogTable { + + private[v2] def validateSchema(cl: Changelog): Unit = { + val byName = cl.columns.map(c => c.name -> c).toMap + def check(name: String, expected: DataType*): Unit = { + val col = byName.getOrElse(name, + throw QueryCompilationErrors.changelogMissingColumnError(cl.name, name)) + if (expected.nonEmpty && col.dataType != expected.head) { + throw QueryCompilationErrors.changelogInvalidColumnTypeError( + cl.name, name, expected.head.sql, col.dataType.sql) + } + } + check("_change_type", StringType) + check("_commit_version") // connector-defined, any type accepted + check("_commit_timestamp", TimestampType) + + // Only call `rowId()` / `rowVersion()` when a capability requires them; a connector + // that advertises a capability without overriding the method surfaces the default + // UnsupportedOperationException directly. + val needsRowId = cl.containsCarryoverRows() || + cl.representsUpdateAsDeleteAndInsert() || + cl.containsIntermediateChanges() + if (needsRowId) { + val rowIds = cl.rowId() + if (rowIds == null || rowIds.isEmpty) { + throw QueryCompilationErrors.changelogMissingRowIdError(cl.name) + } + } + val needsRowVersion = cl.containsCarryoverRows() || + cl.representsUpdateAsDeleteAndInsert() + if (needsRowVersion) { + cl.rowVersion() + } + } +} diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala index 9622d23122318..c56f0e14417e0 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogEndToEndSuite.scala @@ -25,6 +25,7 @@ import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.analysis.NamedStreamingRelation import org.apache.spark.sql.catalyst.streaming.UserProvided import org.apache.spark.sql.connector.catalog._ +import org.apache.spark.sql.connector.catalog.Changelog.{CHANGE_TYPE_DELETE, CHANGE_TYPE_INSERT} import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.test.SharedSparkSession import org.apache.spark.sql.types.{LongType, StringType} @@ -93,12 +94,12 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() returns change data") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "delete", 2L, 2000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_DELETE, 2L, 2000000L))) val expected = Seq( - Row(1L, "a", "insert", 1L, new Timestamp(1000L)), - Row(2L, "b", "delete", 2L, new Timestamp(2000L))) + Row(1L, "a", CHANGE_TYPE_INSERT, 1L, new Timestamp(1000L)), + Row(2L, "b", CHANGE_TYPE_DELETE, 2L, new Timestamp(2000L))) // DataFrame API checkAnswer( @@ -116,13 +117,13 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() with open-ended version range") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 2L, 2000000L), - makeChangeRow(3L, "c", "insert", 3L, 3000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 2L, 2000000L), + makeChangeRow(3L, "c", CHANGE_TYPE_INSERT, 3L, 3000000L))) val expected = Seq( - Row(2L, "b", "insert", 2L, new Timestamp(2000L)), - Row(3L, "c", "insert", 3L, new Timestamp(3000L))) + Row(2L, "b", CHANGE_TYPE_INSERT, 2L, new Timestamp(2000L)), + Row(3L, "c", CHANGE_TYPE_INSERT, 3L, new Timestamp(3000L))) // DataFrame API checkAnswer( @@ -157,12 +158,12 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() select CDC metadata columns") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "delete", 2L, 2000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_DELETE, 2L, 2000000L))) val expected = Seq( - Row(1L, "insert", 1L), - Row(2L, "delete", 2L)) + Row(1L, CHANGE_TYPE_INSERT, 1L), + Row(2L, CHANGE_TYPE_DELETE, 2L)) // DataFrame API checkAnswer( @@ -179,9 +180,9 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() with projection and filter") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 1L, 1000000L), - makeChangeRow(1L, "a2", "insert", 2L, 2000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(1L, "a2", CHANGE_TYPE_INSERT, 2L, 2000000L))) val expected = Seq(Row(1L, "a2")) @@ -200,13 +201,13 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() with aggregation on change types") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 1L, 1000000L), - makeChangeRow(1L, "a", "delete", 2L, 2000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(1L, "a", CHANGE_TYPE_DELETE, 2L, 2000000L))) val expected = Seq( - Row("insert", 2L), - Row("delete", 1L)) + Row(CHANGE_TYPE_INSERT, 2L), + Row(CHANGE_TYPE_DELETE, 1L)) // DataFrame API checkAnswer( @@ -223,7 +224,7 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("schema includes CDC metadata columns") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L))) // DataFrame API val dfApi = spark.read.option("startingVersion", "1").changes(fullTableName) @@ -242,14 +243,14 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() version range filters correctly") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 2L, 2000000L), - makeChangeRow(3L, "c", "insert", 3L, 3000000L), - makeChangeRow(4L, "d", "insert", 4L, 4000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 2L, 2000000L), + makeChangeRow(3L, "c", CHANGE_TYPE_INSERT, 3L, 3000000L), + makeChangeRow(4L, "d", CHANGE_TYPE_INSERT, 4L, 4000000L))) val expected = Seq( - Row(2L, "b", "insert", 2L, new Timestamp(2000L)), - Row(3L, "c", "insert", 3L, new Timestamp(3000L))) + Row(2L, "b", CHANGE_TYPE_INSERT, 2L, new Timestamp(2000L)), + Row(3L, "c", CHANGE_TYPE_INSERT, 3L, new Timestamp(3000L))) // DataFrame API checkAnswer( @@ -269,14 +270,14 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() default bounds are inclusive") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 2L, 2000000L), - makeChangeRow(3L, "c", "insert", 3L, 3000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 2L, 2000000L), + makeChangeRow(3L, "c", CHANGE_TYPE_INSERT, 3L, 3000000L))) val expected = Seq( - Row(1L, "a", "insert", 1L, new Timestamp(1000L)), - Row(2L, "b", "insert", 2L, new Timestamp(2000L)), - Row(3L, "c", "insert", 3L, new Timestamp(3000L))) + Row(1L, "a", CHANGE_TYPE_INSERT, 1L, new Timestamp(1000L)), + Row(2L, "b", CHANGE_TYPE_INSERT, 2L, new Timestamp(2000L)), + Row(3L, "c", CHANGE_TYPE_INSERT, 3L, new Timestamp(3000L))) // DataFrame API - default (both inclusive) checkAnswer( @@ -300,14 +301,14 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() with startingBoundInclusive=false") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 2L, 2000000L), - makeChangeRow(3L, "c", "insert", 3L, 3000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 2L, 2000000L), + makeChangeRow(3L, "c", CHANGE_TYPE_INSERT, 3L, 3000000L))) // Exclusive start: version 1 excluded, versions 2 and 3 included val expected = Seq( - Row(2L, "b", "insert", 2L, new Timestamp(2000L)), - Row(3L, "c", "insert", 3L, new Timestamp(3000L))) + Row(2L, "b", CHANGE_TYPE_INSERT, 2L, new Timestamp(2000L)), + Row(3L, "c", CHANGE_TYPE_INSERT, 3L, new Timestamp(3000L))) // DataFrame API checkAnswer( @@ -327,14 +328,14 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() with endingBoundInclusive=false") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 2L, 2000000L), - makeChangeRow(3L, "c", "insert", 3L, 3000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 2L, 2000000L), + makeChangeRow(3L, "c", CHANGE_TYPE_INSERT, 3L, 3000000L))) // Exclusive end: versions 1 and 2 included, version 3 excluded val expected = Seq( - Row(1L, "a", "insert", 1L, new Timestamp(1000L)), - Row(2L, "b", "insert", 2L, new Timestamp(2000L))) + Row(1L, "a", CHANGE_TYPE_INSERT, 1L, new Timestamp(1000L)), + Row(2L, "b", CHANGE_TYPE_INSERT, 2L, new Timestamp(2000L))) // DataFrame API checkAnswer( @@ -354,13 +355,13 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() with both bounds exclusive") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 2L, 2000000L), - makeChangeRow(3L, "c", "insert", 3L, 3000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 2L, 2000000L), + makeChangeRow(3L, "c", CHANGE_TYPE_INSERT, 3L, 3000000L))) // Both exclusive: only version 2 included val expected = Seq( - Row(2L, "b", "insert", 2L, new Timestamp(2000L))) + Row(2L, "b", CHANGE_TYPE_INSERT, 2L, new Timestamp(2000L))) // DataFrame API checkAnswer( @@ -383,7 +384,7 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() default deduplication mode is dropCarryovers") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L))) // DataFrame API spark.read.option("startingVersion", "1").changes(fullTableName).collect() @@ -400,7 +401,7 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() with deduplicationMode none") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L))) // DataFrame API spark.read @@ -420,7 +421,7 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() passes computeUpdates to catalog") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L))) // DataFrame API spark.read @@ -440,7 +441,7 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("changes() with timestamp range") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L))) // DataFrame API spark.read @@ -475,14 +476,14 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("streaming changes() returns change data") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 1L, 1000000L), - makeChangeRow(1L, "a", "delete", 2L, 2000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(1L, "a", CHANGE_TYPE_DELETE, 2L, 2000000L))) val expected = Seq( - Row(1L, "a", "insert", 1L, new Timestamp(1000L)), - Row(2L, "b", "insert", 1L, new Timestamp(1000L)), - Row(1L, "a", "delete", 2L, new Timestamp(2000L))) + Row(1L, "a", CHANGE_TYPE_INSERT, 1L, new Timestamp(1000L)), + Row(2L, "b", CHANGE_TYPE_INSERT, 1L, new Timestamp(1000L)), + Row(1L, "a", CHANGE_TYPE_DELETE, 2L, new Timestamp(2000L))) // DataFrame API val dfApiStream = spark.readStream @@ -512,12 +513,12 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("streaming changes() with startingVersion filters data") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 1L, 1000000L), - makeChangeRow(1L, "a", "delete", 2L, 2000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(1L, "a", CHANGE_TYPE_DELETE, 2L, 2000000L))) val expected = Seq( - Row(1L, "a", "delete", 2L, new Timestamp(2000L))) + Row(1L, "a", CHANGE_TYPE_DELETE, 2L, new Timestamp(2000L))) // DataFrame API val dfApiStream = spark.readStream @@ -547,9 +548,9 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("streaming changes() with projection and filter") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L), - makeChangeRow(2L, "b", "insert", 1L, 1000000L), - makeChangeRow(3L, "c", "insert", 2L, 2000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(2L, "b", CHANGE_TYPE_INSERT, 1L, 1000000L), + makeChangeRow(3L, "c", CHANGE_TYPE_INSERT, 2L, 2000000L))) val expected = Seq(Row(1L, "a"), Row(2L, "b")) @@ -586,7 +587,7 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("streaming changes() passes computeUpdates to catalog") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L))) // DataFrame API val dfApiStream = spark.readStream @@ -620,10 +621,10 @@ class ChangelogEndToEndSuite extends SharedSparkSession { test("streaming changes() supports .name() API with source evolution enabled") { catalog.addChangeRows(ident, Seq( - makeChangeRow(1L, "a", "insert", 1L, 1000000L))) + makeChangeRow(1L, "a", CHANGE_TYPE_INSERT, 1L, 1000000L))) val expected = Seq( - Row(1L, "a", "insert", 1L, new Timestamp(1000L))) + Row(1L, "a", CHANGE_TYPE_INSERT, 1L, new Timestamp(1000L))) withSQLConf(SQLConf.ENABLE_STREAMING_SOURCE_EVOLUTION.key -> "true") { val stream = spark.readStream diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala index d403db1e62bf9..8f29df44538b6 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/ChangelogResolutionSuite.scala @@ -24,10 +24,12 @@ import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 import org.apache.spark.sql.connector.catalog._ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ import org.apache.spark.sql.connector.catalog.ChangelogRange -import org.apache.spark.sql.connector.expressions.Transform +import org.apache.spark.sql.connector.expressions.{FieldReference, NamedReference, Transform} +import org.apache.spark.sql.connector.read.ScanBuilder import org.apache.spark.sql.execution.datasources.v2.{ChangelogTable, DataSourceV2Relation} import org.apache.spark.sql.test.SharedSparkSession -import org.apache.spark.sql.types.{LongType, StringType} +import org.apache.spark.sql.types.{IntegerType, LongType, StringType, TimestampType} +import org.apache.spark.sql.util.CaseInsensitiveStringMap /** * Tests for the CDC (Change Data Capture) analyzer resolution path: @@ -269,4 +271,201 @@ class ChangelogResolutionSuite extends SharedSparkSession { condition = "INVALID_CDC_OPTION.STREAMING_POST_PROCESSING_NOT_SUPPORTED", parameters = Map("changelogName" -> s"$cdcCatalogName.test_table_changelog")) } + + // =========================================================================== + // Generic changelog schema validation + // =========================================================================== + + private def stubInfo(): ChangelogInfo = new ChangelogInfo( + new ChangelogRange.VersionRange("1", java.util.Optional.of("2"), true, true), + ChangelogInfo.DeduplicationMode.DROP_CARRYOVERS, + false) + + private def cl(name: String, cols: (String, org.apache.spark.sql.types.DataType)*) + : TestChangelog = { + new TestChangelog(name, cols.map { case (n, t) => Column.create(n, t) }.toArray) + } + + private def missing(columnName: String): Map[String, String] = + Map("changelogName" -> "bad_cl", "columnName" -> columnName) + + private def wrongType(columnName: String, expected: String, actual: String) + : Map[String, String] = Map( + "changelogName" -> "bad_cl", + "columnName" -> columnName, + "expectedType" -> expected, + "actualType" -> actual) + + // Valid metadata tuples; tests swap one of these out to create broken schemas. + private val validChangeType = "_change_type" -> StringType + private val validVersion = "_commit_version" -> LongType + private val validTimestamp = "_commit_timestamp" -> TimestampType + + test("ChangelogTable - missing _change_type column throws") { + checkError( + intercept[AnalysisException] { + ChangelogTable(cl("bad_cl", validVersion, validTimestamp), stubInfo()) + }, + condition = "INVALID_CHANGELOG_SCHEMA.MISSING_COLUMN", + parameters = missing("_change_type")) + } + + test("ChangelogTable - missing _commit_version column throws") { + checkError( + intercept[AnalysisException] { + ChangelogTable(cl("bad_cl", validChangeType, validTimestamp), stubInfo()) + }, + condition = "INVALID_CHANGELOG_SCHEMA.MISSING_COLUMN", + parameters = missing("_commit_version")) + } + + test("ChangelogTable - missing _commit_timestamp column throws") { + checkError( + intercept[AnalysisException] { + ChangelogTable(cl("bad_cl", validChangeType, validVersion), stubInfo()) + }, + condition = "INVALID_CHANGELOG_SCHEMA.MISSING_COLUMN", + parameters = missing("_commit_timestamp")) + } + + test("ChangelogTable - wrong _change_type data type throws") { + checkError( + intercept[AnalysisException] { + ChangelogTable( + cl("bad_cl", "_change_type" -> IntegerType, validVersion, validTimestamp), + stubInfo()) + }, + condition = "INVALID_CHANGELOG_SCHEMA.INVALID_COLUMN_TYPE", + parameters = wrongType("_change_type", "STRING", "INT")) + } + + test("ChangelogTable - wrong _commit_timestamp data type throws") { + checkError( + intercept[AnalysisException] { + ChangelogTable( + cl("bad_cl", validChangeType, validVersion, "_commit_timestamp" -> LongType), + stubInfo()) + }, + condition = "INVALID_CHANGELOG_SCHEMA.INVALID_COLUMN_TYPE", + parameters = wrongType("_commit_timestamp", "TIMESTAMP", "BIGINT")) + } + + test("ChangelogTable - _commit_version type is connector-defined (any type accepted)") { + Seq(IntegerType, LongType, StringType).foreach { versionType => + ChangelogTable( + cl("any_cl", validChangeType, "_commit_version" -> versionType, validTimestamp), + stubInfo()) + } + } + + test("ChangelogTable - valid schema with data columns passes") { + ChangelogTable( + cl("good_cl", "id" -> LongType, "name" -> StringType, + validChangeType, validVersion, validTimestamp), + stubInfo()) + } + + test("ChangelogTable - nested rowId and rowVersion references pass (Delta-style _metadata)") { + val metadataRowId = FieldReference(Seq("_metadata", "row_id")) + val metadataRowVersion = FieldReference(Seq("_metadata", "row_commit_version")) + val cl = new TestChangelog( + "delta_cl", + Array( + Column.create("id", LongType, false), + Column.create("_change_type", StringType), + Column.create("_commit_version", LongType), + Column.create("_commit_timestamp", TimestampType)), + carryoverRows = true, + rowIdRefs = Array(metadataRowId), + rowVersionRef = Some(metadataRowVersion)) + ChangelogTable(cl, stubInfo()) + } + + test("ChangelogTable - representsUpdateAsDeleteAndInsert=true requires non-empty rowId") { + val cl = new TestChangelog( + "bad_cl", + Array( + Column.create("_change_type", StringType), + Column.create("_commit_version", LongType), + Column.create("_commit_timestamp", TimestampType)), + updateAsDeleteInsert = true, + rowIdRefs = Array.empty, + rowVersionRef = Some(FieldReference.column("_commit_version"))) + checkError( + intercept[AnalysisException] { ChangelogTable(cl, stubInfo()) }, + condition = "INVALID_CHANGELOG_SCHEMA.MISSING_ROW_ID", + parameters = Map("changelogName" -> "bad_cl")) + } + + test("ChangelogTable - containsIntermediateChanges=true requires non-empty rowId") { + val cl = new TestChangelog( + "bad_cl", + Array( + Column.create("_change_type", StringType), + Column.create("_commit_version", LongType), + Column.create("_commit_timestamp", TimestampType)), + intermediateChanges = true, + rowIdRefs = Array.empty) + checkError( + intercept[AnalysisException] { ChangelogTable(cl, stubInfo()) }, + condition = "INVALID_CHANGELOG_SCHEMA.MISSING_ROW_ID", + parameters = Map("changelogName" -> "bad_cl")) + } + + test("ChangelogTable - UnsupportedOperationException surfaces when rowId() not implemented") { + val cl = new TestChangelog( + "bad_cl", + Array( + Column.create("_change_type", StringType), + Column.create("_commit_version", LongType), + Column.create("_commit_timestamp", TimestampType)), + carryoverRows = true, + rowIdSupported = false, + rowVersionRef = Some(FieldReference.column("_commit_version"))) + intercept[UnsupportedOperationException] { ChangelogTable(cl, stubInfo()) } + } + + test("ChangelogTable - UnsupportedOperationException surfaces when rowVersion() missing") { + val cl = new TestChangelog( + "bad_cl", + Array( + Column.create("_change_type", StringType), + Column.create("_commit_version", LongType), + Column.create("_commit_timestamp", TimestampType)), + carryoverRows = true, + rowIdRefs = Array(FieldReference.column("id")), + rowVersionRef = None) + intercept[UnsupportedOperationException] { ChangelogTable(cl, stubInfo()) } + } + +} + +/** + * Test-only [[Changelog]] implementation that returns a hand-crafted schema. Used to + * exercise [[ChangelogTable]]'s schema validation without going through a real catalog. + * + * Defaults match a minimal connector with no post-processing capabilities. Tests opt + * into capability flags or `rowVersion()` overrides via constructor params. + */ +private class TestChangelog( + nameArg: String, + cols: Array[Column], + carryoverRows: Boolean = false, + updateAsDeleteInsert: Boolean = false, + intermediateChanges: Boolean = false, + rowIdRefs: Array[NamedReference] = Array.empty, + rowIdSupported: Boolean = true, + rowVersionRef: Option[NamedReference] = None) extends Changelog { + override def name(): String = nameArg + override def columns(): Array[Column] = cols + override def containsCarryoverRows(): Boolean = carryoverRows + override def containsIntermediateChanges(): Boolean = intermediateChanges + override def representsUpdateAsDeleteAndInsert(): Boolean = updateAsDeleteInsert + override def rowId(): Array[NamedReference] = + if (rowIdSupported) rowIdRefs else super.rowId() + override def rowVersion(): NamedReference = + rowVersionRef.getOrElse(super.rowVersion()) + override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = { + throw new UnsupportedOperationException("not needed for schema validation tests") + } } From 68456a6aa686dc32eda0a6a2cd1a2c9e841762eb Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Tue, 28 Apr 2026 22:09:08 -0700 Subject: [PATCH 013/286] [SPARK-56653][K8S][DOCS] Document `spark.kubernetes.(executor.useDriverPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)` ### What changes were proposed in this pull request? This PR documents three missing Spark configurations introduced in Spark 4.1.0. | Property | Default | Since | Introduced in | | --- | --- | --- | --- | | `spark.kubernetes.executor.useDriverPodIP` | `false` | 4.1.0 | SPARK-53944 | | `spark.kubernetes.driver.annotateExitException` | `false` | 4.1.0 | SPARK-53335 | | `spark.kubernetes.allocation.maxPendingPodsPerRp` | `Int.MaxValue` | 4.1.0 | SPARK-53324 | ### Why are the changes needed? Three user-facing Kubernetes configurations added in Spark 4.1.0 but weren't documented yet. - https://github.com/apache/spark/pull/52650 - https://github.com/apache/spark/pull/52068 - https://github.com/apache/spark/pull/51913 ### Does this PR introduce _any_ user-facing change? No. Documentation-only update. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes #55591 from dongjoon-hyun/SPARK-56653. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- docs/running-on-kubernetes.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index 5aa906f9f37d9..b575a127199cb 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -1557,6 +1557,14 @@ See the [configuration page](configuration.html) for information on Spark config 3.2.0 + + spark.kubernetes.driver.annotateExitException + false + + If set to true, Spark will store the exit exception failed applications in the Kubernetes API server using the spark.exit-exception annotation. + + 4.1.0 + spark.kubernetes.driver.service.ipFamilyPolicy SingleStack @@ -1575,6 +1583,14 @@ See the [configuration page](configuration.html) for information on Spark config 3.4.0 + + spark.kubernetes.executor.useDriverPodIP + false + + If true, executor pods use Driver pod IP directly instead of Driver Service. + + 4.1.0 + spark.kubernetes.driver.ownPersistentVolumeClaim true @@ -1672,6 +1688,17 @@ See the [configuration page](configuration.html) for information on Spark config 3.2.0 + + spark.kubernetes.allocation.maxPendingPodsPerRp + Int.MaxValue + + Maximum number of pending PODs allowed per resource profile ID during executor + allocation. This provides finer-grained control over pending pods by limiting them + per resource profile rather than globally. When set, this limit is enforced + independently for each resource profile ID. + + 4.1.0 + spark.kubernetes.allocation.pods.allocator direct From 661afd4d8a17175597aa7bfb5cf721fa62de55e0 Mon Sep 17 00:00:00 2001 From: Ruifeng Zheng Date: Wed, 29 Apr 2026 18:20:12 +0800 Subject: [PATCH 014/286] [SPARK-56614][SQL][CONNECT][TESTS][FOLLOWUP] Pin strictDataFrameColumnResolution=true for lazy column validation test ### What changes were proposed in this pull request? Pin `spark.sql.analyzer.strictDataFrameColumnResolution=true` around the body of the `lazy column validation` test in `DataFrameSuite`. The config is set via `spark.conf.set/unset` rather than `withSQLConf` because the lazy SQLConf entry trips `withSQLConf`'s `isModifiable` check on the Connect server. ### Why are the changes needed? The test asserts that `df4.schema` throws `AnalysisException` for `df1("x")` when `df1` does not contain `x`. This holds only under strict plan-id-based resolution; if the name-based fallback path is enabled, `df1("x")` resolves to `df2.x` from the join output and `df4.schema` succeeds. Today this works because `STRICT_DATAFRAME_COLUMN_RESOLUTION` defaults to `true`, but the test should not silently rely on that default; pinning it makes the assumption explicit and keeps the test robust against future default changes or environments where the default is overridden. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? Existing `DataFrameSuite."lazy column validation"`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Anthropic), claude-opus-4-7 Closes #55604 from zhengruifeng/SPARK-strict-df-col-resolution-test-pin. Authored-by: Ruifeng Zheng Signed-off-by: Ruifeng Zheng --- .../spark/sql/connect/DataFrameSuite.scala | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/DataFrameSuite.scala b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/DataFrameSuite.scala index 890245fdd2fba..57b8080c4b137 100644 --- a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/DataFrameSuite.scala +++ b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/DataFrameSuite.scala @@ -68,14 +68,24 @@ class DataFrameSuite extends QueryTest with RemoteSparkSession { } test("lazy column validation") { - val session = spark - import session.implicits._ - - val df1 = Seq(1 -> "y").toDF("a", "y") - val df2 = Seq(1 -> "x").toDF("a", "x") - val df3 = df1.join(df2, df1("a") === df2("a")) - val df4 = df3.select(df1("x")) // <- No exception here - - intercept[AnalysisException] { df4.schema } + // The test relies on strict plan-id-based resolution: with the name-based fallback + // enabled, df1("x") would resolve to df2.x via the join output and df4.schema would + // succeed. Pin the config directly via spark.conf.set/unset; the lazy SQLConf entry + // trips withSQLConf's isModifiable check on the Connect server, so we cannot use that + // helper here. + spark.conf.set("spark.sql.analyzer.strictDataFrameColumnResolution", "true") + try { + val session = spark + import session.implicits._ + + val df1 = Seq(1 -> "y").toDF("a", "y") + val df2 = Seq(1 -> "x").toDF("a", "x") + val df3 = df1.join(df2, df1("a") === df2("a")) + val df4 = df3.select(df1("x")) // <- No exception here + + intercept[AnalysisException] { df4.schema } + } finally { + spark.conf.unset("spark.sql.analyzer.strictDataFrameColumnResolution") + } } } From 82ad46f3bb4cfa3a08921beb77fd45d421133be1 Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Wed, 29 Apr 2026 19:33:06 +0800 Subject: [PATCH 015/286] [SPARK-56636][INFRA] Decouple scalastyle from compile + clean up unidoc log noise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? Three related changes that fix the most common CI debug pain on this project: a single root cause cascading into many red checks, each surfacing only "Process completed with exit code 1" with no file:line. **1. Structural decoupling — style is no longer attached to compile.** Today, scalastyle runs as a side effect of `(Compile / compile)` in SBT and as the default `` of `scalastyle-maven-plugin` bound to the Maven `verify` phase. A single style violation in `core` aborts the compile of `core` and every transitive dependent, so every CI job that recompiles those sources fails — Build modules, Documentation generation, Java 17/25 Maven build, sparkr, Docker integration tests, TPC-DS — each with no file/line in its annotation. The dedicated lint job is just one of seven jobs that fail; nothing tells the user it's the root cause. This PR removes the coupling at both build systems: - `project/SparkBuild.scala`: drop the `(Compile / compile) := { scalaStyleOnCompile.value; … }` (and matching `(Test / compile)`) hooks from `enableScalaStyle`. The standalone `scalaStyleOnCompile` / `scalaStyleOnTest` task definitions remain so `dev/lint-scala` → `dev/scalastyle` → `sbt scalastyle test:scalastyle` continues to work; only the compile-time auto-invocation is removed. `NOLINT_ON_COMPILE` env-var gating is dropped (no-op once the hook is gone). - `pom.xml`: remove the default `` from `scalastyle-maven-plugin` and add an opt-in `scalastyle` profile that re-binds `check`. Default Maven builds (`mvn install`, `mvn package`, etc.) no longer trigger scalastyle; activate the profile (`mvn ... -Pscalastyle`) or invoke the goal directly (`mvn scalastyle:check`) to run it. - `.github/workflows/build_and_test.yml`: remove the now-no-op `NOLINT_ON_COMPILE` env vars from Build modules / pyspark / sparkr / lint / docs jobs. After this change, scalastyle runs in exactly one place in CI: the dedicated lint job (via SBT through `dev/lint-scala`). A style violation fails ONE check; every other CI job stays green because they don't see the violation. No `needs: lint` chaining is needed — the cascade is gone at the source. Style coverage is unchanged (the lint job still scans every Scala file with the same config). **2. Surface scalastyle violations as inline PR annotations.** Even with the cascade removed, the lint job's natural output is verbose sbt output where the actual file:line of a violation is buried. `dev/scalastyle` now parses scalastyle's output and re-emits each violation as a GitHub Actions `::error file=...,line=...,title=Scalastyle::...` annotation when running under CI. The violation appears inline on the PR's "Files changed" tab — no log scrolling needed. Two output formats are handled: - scalastyle's native console writer: `error file= message= line= [column=]`, used by the explicit `scalastyle` / `test:scalastyle` tasks. - sbt's logger format: `[error] :: `, used when `Tasks.doScalastyle` is invoked through sbt's logger (the format we actually see in CI today). The two regex branches are precise enough to skip near-miss lines: sbt's "task failed" summaries, exception traces, and regular Scala compile errors with their `:LINE:COL:` triple-colon shape are correctly not matched. **3. Strip genjavadoc-stub diagnostic noise from the CI log.** `docs/_plugins/build_api_docs.rb` wraps the `build/sbt unidoc` invocation with a small inline state machine that drops genjavadoc-stub `[error]` blocks from CI stdout. javadoc emits ~3500 such lines per unidoc run (`cannot find symbol` / `illegal combination of modifiers` / `non-static type variable` / `X is not public in org.apache.spark.Y; cannot be accessed from outside package` on stub `T1, T2, ...` type variables and stub references to package-private Scala types) — all benign because `--ignore-source-errors` is set, but they bury everything else. Each diagnostic is a header line followed by 3–5 `[error|warn]`-prefixed continuation lines (snippet, caret, symbol/location); the state machine drops both. The filter matches by **message text**, not just by `target/java/` path, so legitimate doclint diagnostics on stub paths (e.g. a heading-out-of-sequence in a Scala doc comment that genjavadoc preserves into the stub) survive. Real `src/main/java/` diagnostics are untouched. Note: this PR previously also experimented with `-Xdoclint:html` and a custom doclet to surface the per-file diagnostic when unidoc fails. That work was dropped after PR #55581 (https://github.com/apache/spark/pull/55581) identified the real root cause: javadoc's default `-Xmaxerrs 100` caps error reporting at 100, so the diagnostics we wanted never reached our interception layer at all. PR #55581's `-Xmaxerrs`/`-Xmaxwarns` bump and `-verbose` addition is the right fix and is complementary to the log filter shipped here. ### Why are the changes needed? A single scalastyle violation in catalyst recently caused 7 red checks at once, each surfacing only a generic "exit code 1" annotation; finding the actual file:line required grepping through a multi-megabyte job log. Decoupling collapses the style cascade to one red check; the scalastyle annotation makes that check actionable on the PR's Files-changed tab; the genjavadoc-stub log filter cuts ~3500 lines of inert noise from every unidoc run so the rest of the doc-gen output stays scannable. ### Does this PR introduce _any_ user-facing change? No SQL or runtime changes. Two developer-facing changes: - `mvn install` (or any Maven invocation that hits the `verify` phase) no longer runs scalastyle by default. Activate the `scalastyle` profile (`mvn ... -Pscalastyle`) or invoke `mvn scalastyle:check` explicitly. The lint job in CI continues to enforce style. - `sbt compile` no longer triggers scalastyle. Run `sbt scalastyle test:scalastyle` (or `dev/lint-scala`) explicitly to check style locally; CI still enforces it. ### How was this patch tested? - **Annotation parsers** were unit-checked locally against representative real CI samples for both scalastyle output formats (native + sbt logger). Near-miss lines that must NOT match (sbt task-failed summaries, exception traces, compile errors with the `:LINE:COL:` shape) are correctly skipped. - **End-to-end annotation rendering** confirmed on an earlier CI run by planting a deliberate scalastyle violation on this branch (later reverted): the violation surfaced as an inline annotation on the PR's "Files changed" tab pointing at the right file:line. - **Stub-line filter** unit-checked locally against representative samples for the four known-benign genjavadoc structural error patterns plus two doclint diagnostics on stub paths plus three non-stub control cases — all classify correctly. End-to-end verified by inspecting captured CI logs from earlier failed unidoc runs on this branch. - **Style coverage unchanged**: the lint job continues to run `sbt scalastyle test:scalastyle` exactly as before; the same scalastyle config (`scalastyle-config.xml`) governs which violations fire. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (Anthropic) Closes #55563 from cloud-fan/ci-error-clarity. Authored-by: Wenchen Fan Signed-off-by: Wenchen Fan --- .github/workflows/build_and_test.yml | 4 - dev/make-distribution.sh | 1 - dev/scalastyle | 56 ++++++++++++ docs/_plugins/build_api_docs.rb | 125 ++++++++------------------- pom.xml | 38 ++++++-- project/SparkBuild.scala | 25 +++--- 6 files changed, 135 insertions(+), 114 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 8dc6303a81239..dd736a7191598 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -337,7 +337,6 @@ jobs: HIVE_PROFILE: ${{ matrix.hive }} GITHUB_PREV_SHA: ${{ github.event.before }} SPARK_LOCAL_IP: localhost - NOLINT_ON_COMPILE: true SKIP_UNIDOC: true SKIP_MIMA: true SKIP_PACKAGING: true @@ -599,7 +598,6 @@ jobs: HIVE_PROFILE: hive2.3 GITHUB_PREV_SHA: ${{ github.event.before }} SPARK_LOCAL_IP: localhost - NOLINT_ON_COMPILE: true SKIP_UNIDOC: true SKIP_MIMA: true SKIP_PACKAGING: true @@ -868,7 +866,6 @@ jobs: env: LC_ALL: C.UTF-8 LANG: C.UTF-8 - NOLINT_ON_COMPILE: false GITHUB_PREV_SHA: ${{ github.event.before }} BRANCH: ${{ inputs.branch }} container: @@ -1060,7 +1057,6 @@ jobs: env: LC_ALL: C.UTF-8 LANG: C.UTF-8 - NOLINT_ON_COMPILE: false PYSPARK_DRIVER_PYTHON: python3.9 PYSPARK_PYTHON: python3.9 GITHUB_PREV_SHA: ${{ github.event.before }} diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh index 16598bda87339..428a3ed3f1120 100755 --- a/dev/make-distribution.sh +++ b/dev/make-distribution.sh @@ -169,7 +169,6 @@ fi cd "$SPARK_HOME" if [ "$SBT_ENABLED" == "true" ] ; then - export NOLINT_ON_COMPILE=1 # Store the command as an array because $SBT variable might have spaces in it. # Normal quoting tricks don't work. # See: http://mywiki.wooledge.org/BashFAQ/050 diff --git a/dev/scalastyle b/dev/scalastyle index 0428453b62c81..09e6c2372614d 100755 --- a/dev/scalastyle +++ b/dev/scalastyle @@ -30,6 +30,62 @@ ERRORS=$(echo -e "q\n" \ if test ! -z "$ERRORS"; then echo -e "Scalastyle checks failed at following occurrences:\n$ERRORS" + # When running under GitHub Actions, also emit each scalastyle violation as + # a workflow `::error` annotation so it appears inline on the PR's "Files + # changed" tab. Without this, a violation cascades into ~7 red CI checks + # (Linters, Java 17/25 Maven build, Documentation generation, sparkr, + # Docker integration, TPC-DS) -- all needing catalyst to compile -- and + # each only surfaces a generic "exit code 1" with no file/line, forcing + # the user to download a full job log to find the actual violation. + if [[ "${GITHUB_ACTIONS:-}" == "true" ]]; then + # Strip ANSI color codes from the captured output before regex + # matching. Today sbt under awk's pipe is not a TTY and skips color, + # so the input is already plain. But if sbt color is ever forced + # (`-Dsbt.color=always`, custom CI shell), `\e[31m` would silently + # break every regex below. Cheap to harden. + ERRORS_PLAIN=$(printf '%s' "$ERRORS" | sed -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g') + # Helper: emit one `::error` annotation. Centralised so the two regex + # branches below stay short. + emit_annotation() { + local file="$1" lineno="$2" msg="$3" + # Strip the GitHub Actions workspace prefix so the annotation + # references the path as it appears in the repo. + local file_rel="${file#${GITHUB_WORKSPACE:-}/}" + # Escape the few characters GitHub reserves in annotation values: + # %, \r, \n. (`,` and `:` need not be escaped in the message body, + # only inside parameter values, which we don't use.) + local msg_escaped="${msg//%/%25}" + msg_escaped="${msg_escaped//$'\r'/%0D}" + msg_escaped="${msg_escaped//$'\n'/%0A}" + printf '::error file=%s,line=%s,title=Scalastyle::%s\n' \ + "$file_rel" "$lineno" "$msg_escaped" + } + printf '%s\n' "$ERRORS_PLAIN" | while IFS= read -r raw; do + # Two scalastyle output formats reach us: + # + # (a) scalastyle's native console writer (`Tasks.doScalastyle` when + # invoked by the explicit `scalastyle` / `test:scalastyle` + # tasks): + # error file= message= line= [column=] + # The path has no spaces, the message can; `column=` is + # appended for checkers that report a column (e.g. + # `WhitespaceEndOfLineChecker`) and absent otherwise. + # + # (b) sbt's logger format, used when `Tasks.doScalastyle` writes + # through `streams.value.log.error(...)` -- which is what the + # explicit `scalastyle` / `test:scalastyle` tasks invoked by + # this script do, and so this is the format we see in CI: + # [error] :: + # The leading `[error] ` plus a single `::` (with no + # `::` follow-up) is what tells it apart from a regular + # Scala compile error of shape `[error] ::: `. + if [[ "$raw" =~ ^error[[:space:]]+file=([^[:space:]]+)[[:space:]]+message=(.*)[[:space:]]+line=([0-9]+)([[:space:]]+column=[0-9]+)?$ ]]; then + emit_annotation "${BASH_REMATCH[1]}" "${BASH_REMATCH[3]}" "${BASH_REMATCH[2]}" + elif [[ "$raw" =~ ^\[error\][[:space:]]+(/[^:[:space:]]+):([0-9]+):[[:space:]]+(.+)$ ]]; then + emit_annotation "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}" "${BASH_REMATCH[3]}" + fi + done + fi exit 1 else echo -e "Scalastyle checks passed." diff --git a/docs/_plugins/build_api_docs.rb b/docs/_plugins/build_api_docs.rb index e6719c4bed7e3..a75727d8b65db 100644 --- a/docs/_plugins/build_api_docs.rb +++ b/docs/_plugins/build_api_docs.rb @@ -133,101 +133,48 @@ def build_spark_scala_and_java_docs_if_necessary command = "build/sbt -Pkinesis-asl unidoc" puts "Running '#{command}'..." - # Tee sbt output to a log file so we can diagnose failures. The most common - # unidoc failure is a javadoc crash mid-stream while generating HTML for a - # specific class, buried under ~100 benign errors on genjavadoc-generated - # Java stubs (e.g. target/java/org/apache/spark/ErrorInfo.java). Without the - # diagnostic below, the real culprit -- the source whose doc tripped javadoc - # -- is effectively invisible in the CI log. - log_file = File.join(SPARK_PROJECT_ROOT, "target", "unidoc-build.log") - mkdir_p(File.dirname(log_file)) - success = stream_and_capture(command, log_file) - unless success - diagnose_unidoc_failure(log_file) - raise("Unidoc generation failed") - end -end -# Runs `command`, streaming every line to both stdout and `log_file`. Returns -# true iff the command exited 0. Ruby-only; no shell pipefail reliance. -def stream_and_capture(command, log_file) - File.open(log_file, 'w') do |f| - IO.popen("#{command} 2>&1", 'r') do |pipe| - pipe.each_line do |line| + # Suppress genjavadoc-stub diagnostic blocks from the visible log. javadoc + # emits ~3500 `[error]` lines per unidoc run on stubs under `target/java/` + # -- all benign because `--ignore-source-errors` is set, but they bury + # everything else. Each diagnostic is a header line followed by 3-5 + # `[error|warn]`-prefixed continuation lines (snippet, caret, + # symbol/location); the state machine drops both. + # + # Match by *message text*, not just by `target/java/` path. Otherwise + # legitimate doclint diagnostics on stub paths would be hidden too -- + # those messages are real signal. The patterns below are the known-benign + # genjavadoc structural errors; anything else on a `target/java/` path is + # echoed. Diagnostic mirror lines from `SparkUnidocDoclet` use the + # `[unidoc-doclet]` prefix and don't match either regex, so they always + # pass through. + ansi = /\e\[[0-9;]*[A-Za-z]/ + stub_header = %r{ + \[(?:error|warn)\]\s+ + \S*?/target/java/\S+\.java:\d+(?::\d+)?:\s+ + error:\s+ + (?:cannot\s+find\s+symbol + |illegal\s+combination\s+of\s+modifiers + |non-static\s+type\s+variable\b + |.*?\s+is\s+not\s+public\s+in\s+\S+;\s+cannot\s+be\s+accessed\s+from\s+outside\s+package) + }x + stub_cont = %r{\A\s*\[(?:error|warn)\]\s+(?!/\S+\.java:\d+(?::\d+)?:\s)} + in_stub = false + IO.popen("#{command} 2>&1", 'r') do |pipe| + pipe.each_line do |line| + plain = line.gsub(ansi, '') + if plain =~ stub_header + in_stub = true + elsif in_stub && plain =~ stub_cont + # continuation of a stub block; suppress + else + in_stub = false $stdout.write(line) $stdout.flush - f.write(line) end end end - $?.success? -end - -# Scans the captured unidoc log and prints a pointer to the most likely -# culprit source file. The heuristic: when javadoc dies mid-HTML-generation, -# the last "Generating .../X.html" line before "javadoc exited with exit code" -# names the class that tripped it. Prints nothing actionable if the failure -# mode doesn't match (e.g. a scaladoc error), in which case the full log above -# already shows what's wrong. -def diagnose_unidoc_failure(log_file) - return unless File.exist?(log_file) - begin - lines = File.readlines(log_file) - - javadoc_exit_idx = lines.rindex { |l| l.include?("javadoc exited with exit code") } - last_generating = nil - if javadoc_exit_idx - # Strip ANSI color codes so the regex matches sbt-coloured output too. - ansi = /\e\[[0-9;]*[A-Za-z]/ - lines[0...javadoc_exit_idx].reverse_each do |line| - if line.gsub(ansi, '') =~ %r{Generating .+/javaunidoc/(\S+?\.html)\.\.\.} - last_generating = $1 - break - end - end - end - - banner = "=" * 78 - $stderr.puts "" - $stderr.puts banner - $stderr.puts "Unidoc failed -- diagnostic summary" - $stderr.puts banner - if last_generating - class_path = last_generating.sub(/\.html$/, '') - class_name = class_path.tr('/', '.') - $stderr.puts "" - $stderr.puts " Javadoc crashed while generating: #{last_generating}" - $stderr.puts " Likely culprit: doc comment in #{class_name}" - $stderr.puts "" - $stderr.puts " Javadoc can hard-exit (not just warn) on specific scaladoc" - $stderr.puts " patterns once they have been passed through genjavadoc --" - $stderr.puts " wiki-style `[[Class]]` / `[[method]]` links or inline-backticked" - $stderr.puts " code refs in the Scala source for the class above are common" - $stderr.puts " triggers. Start by auditing any recent doc-string changes in" - $stderr.puts " that source file." - $stderr.puts "" - $stderr.puts " NOTE: the '[error]' lines above on files under" - $stderr.puts " target/java/... are benign genjavadoc stubs -- every PR" - $stderr.puts " emits them and they do not cause the exit. Ignore them." - elsif javadoc_exit_idx - $stderr.puts "" - $stderr.puts " Javadoc exited but no class HTML generation was in progress;" - $stderr.puts " the crash predates HTML output -- likely a CLI / classpath /" - $stderr.puts " setup issue. See the full sbt output above." - else - $stderr.puts "" - $stderr.puts " Could not locate a 'javadoc exited with exit code' marker in" - $stderr.puts " the log; the failure is likely outside the javaunidoc step" - $stderr.puts " (scaladoc / sbt / build env). See the full sbt output above." - end - $stderr.puts banner - $stderr.puts "" - rescue => e - # Never let the diagnostic helper itself obscure the underlying unidoc - # failure: if anything here goes wrong (e.g. encoding error reading the - # log), report it briefly and let the caller raise the real error. - $stderr.puts "(diagnostic helper failed: #{e.class}: #{e.message})" - end + raise("Unidoc generation failed") unless $?.success? end def build_scala_and_java_docs diff --git a/pom.xml b/pom.xml index 4d0158a6928d4..d7dac399c2aed 100644 --- a/pom.xml +++ b/pom.xml @@ -3235,6 +3235,11 @@ org.apache.maven.plugins maven-source-plugin + org.scalastyle scalastyle-maven-plugin @@ -3251,13 +3256,6 @@ ${project.build.sourceEncoding} ${project.reporting.outputEncoding} - - - - check - - - org.apache.maven.plugins @@ -3395,6 +3393,32 @@ + + + scalastyle + + + + + org.scalastyle + scalastyle-maven-plugin + + + + check + + + + + + + + + 6.2.0 4.2.12.Final 2.0.76.Final diff --git a/sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java b/sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java new file mode 100644 index 0000000000000..0ec73caca72ae --- /dev/null +++ b/sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java @@ -0,0 +1,561 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datasketches.memory.internal; + +import static org.apache.datasketches.memory.internal.UnsafeUtil.unsafe; +import static org.apache.datasketches.memory.internal.Util.characterPad; + +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.datasketches.memory.MemoryBoundsException; +import org.apache.datasketches.memory.MemoryRequestServer; +import org.apache.datasketches.memory.ReadOnlyException; +import org.apache.datasketches.memory.Resource; + +// Ported from https://github.com/apache/datasketches-memory/pull/272 +// which relaxes the `checkJavaVersion` to allow Java 25 + +/** + * Implements the root Resource methods plus some common static variables and check methods. + * + * @author Lee Rhodes + */ +@SuppressWarnings("restriction") +public abstract class ResourceImpl implements Resource { + static final String JDK; + static final int JDK_MAJOR; //8, 11, 17, etc + + //Used to convert "type" to bytes: bytes = longs << LONG_SHIFT + static final int BOOLEAN_SHIFT = 0; + static final int BYTE_SHIFT = 0; + static final long SHORT_SHIFT = 1; + static final long CHAR_SHIFT = 1; + static final long INT_SHIFT = 2; + static final long LONG_SHIFT = 3; + static final long FLOAT_SHIFT = 2; + static final long DOUBLE_SHIFT = 3; + + //class type IDs. Do not change the bit orders + //The lowest 3 bits are set dynamically + // 0000 0XXX Group 1 + static final int WRITABLE = 0; //bit 0 = 0 + static final int READONLY = 1; //bit 0 + static final int REGION = 2; //bit 1 + static final int DUPLICATE = 4; //bit 2, for Buffer only + + // 000X X000 Group 2 + static final int HEAP = 0; //bits 3,4 = 0 + static final int DIRECT = 8; //bit 3 + static final int MAP = 16; //bit 4, Map is effectively Direct + + // 00X0 0000 Group 3 ByteOrder + static final int NATIVE_BO = 0; //bit 5 = 0 + static final int NONNATIVE_BO = 32;//bit 5 + + // 0X00 0000 Group 4 + static final int MEMORY = 0; //bit 6 = 0 + static final int BUFFER = 64; //bit 6 + + // X000 0000 Group 5 + static final int BYTEBUF = 128; //bit 7 + + /** + * The java line separator character as a String. + */ + public static final String LS = System.getProperty("line.separator"); + + static final String NOT_MAPPED_FILE_RESOURCE = "This is not a memory-mapped file resource"; + static final String THREAD_EXCEPTION_TEXT = "Attempted access outside owning thread"; + + private static AtomicBoolean JAVA_VERSION_WARNING_PRINTED = new AtomicBoolean(false); + + static { + final String jdkVer = System.getProperty("java.version"); + final int[] p = parseJavaVersion(jdkVer); + JDK = p[0] + "." + p[1]; + JDK_MAJOR = (p[0] == 1) ? p[1] : p[0]; + } + + //set by the leaf nodes + long capacityBytes; + long cumOffsetBytes; + long offsetBytes; + int typeId; + Thread owner = null; + + /** + * The root of the Memory inheritance hierarchy + */ + ResourceImpl() { } + + //MemoryRequestServer logic + + /** + * User specified MemoryRequestServer. Set here and by leaf nodes. + */ + MemoryRequestServer memReqSvr = null; + + @Override + public MemoryRequestServer getMemoryRequestServer() { + return memReqSvr; + } + + @Override + public boolean hasMemoryRequestServer() { + return memReqSvr != null; + } + + @Override + public void setMemoryRequestServer(final MemoryRequestServer memReqSvr) { this.memReqSvr = memReqSvr; } + + //*** + + /** + * Check the requested offset and length against the allocated size. + * The invariants equation is: {@code 0 <= reqOff <= reqLen <= reqOff + reqLen <= allocSize}. + * If this equation is violated an {@link MemoryBoundsException} will be thrown. + * @param reqOff the requested offset + * @param reqLen the requested length + * @param allocSize the allocated size. + * @throws MemoryBoundsException if the given arguments constitute a violation + * of the invariants equation expressed above. + */ + public static void checkBounds(final long reqOff, final long reqLen, final long allocSize) { + if ((reqOff | reqLen | (reqOff + reqLen) | (allocSize - (reqOff + reqLen))) < 0) { + throw new MemoryBoundsException( + "reqOffset: " + reqOff + ", reqLength: " + reqLen + + ", (reqOff + reqLen): " + (reqOff + reqLen) + ", allocSize: " + allocSize); + } + } + + /** + * Checks the runtime Java Version string. Note that Java 17 and 21 is allowed only because some clients do not use the + * WritableMemory.allocateDirect(..) and related functions, which will not work with Java versions >= 14. + * The on-heap functions may work with 17 and 21, nonetheless, versions > Java 11 are not officially supported. + * Caveat emptor. + * @param jdkVer the System.getProperty("java.version") string of the form "p0.p1.X" + * @param p0 The first number group + * @param p1 The second number group + */ + static void checkJavaVersion(final String jdkVer, final int p0, final int p1 ) { + final boolean ok = ((p0 == 1) && (p1 == 8)) || (p0 == 8) || (p0 == 11) || (p0 == 17 || (p0 == 21) || (p0 == 25)); + if (!ok) { throw new IllegalArgumentException( + "Unsupported JDK Major Version. It must be one of 1.8, 8, 11, 17, 21, 25: " + jdkVer); + } + if (p0 > 11 && JAVA_VERSION_WARNING_PRINTED.compareAndSet(false, true)) { + System.err.println( + "Warning: Java versions > Java 11 can only operate in restricted mode where no off-heap operations are allowed!"); + } + } + + void checkNotReadOnly() { + if (isReadOnly()) { + throw new ReadOnlyException("Cannot write to a read-only Resource."); + } + } + + /** + * This checks that the current thread is the same as the given owner thread. + * @Throws IllegalStateException if it is not. + * @param owner the given owner thread. + */ + static final void checkThread(final Thread owner) { + if (owner != Thread.currentThread()) { + throw new IllegalStateException(THREAD_EXCEPTION_TEXT); + } + } + + /** + * @throws IllegalStateException if this Resource is AutoCloseable, and already closed, i.e., not alive. + */ + void checkValid() { + if (!isAlive()) { + throw new IllegalStateException("this Resource is AutoCloseable, and already closed, i.e., not alive."); + } + } + + /** + * Checks that this resource is still valid and throws a MemoryInvalidException if it is not. + * Checks that the specified range of bytes is within bounds of this resource, throws + * {@link MemoryBoundsException} if it's not: i. e. if offsetBytes < 0, or length < 0, + * or offsetBytes + length > {@link #getCapacity()}. + * @param offsetBytes the given offset in bytes of this object + * @param lengthBytes the given length in bytes of this object + * @throws IllegalStateException if this resource is AutoCloseable and is no longer valid, i.e., + * it has already been closed. + * @throws MemoryBoundsException if this resource violates the memory bounds of this resource. + */ + public final void checkValidAndBounds(final long offsetBytes, final long lengthBytes) { + checkValid(); + checkBounds(offsetBytes, lengthBytes, getCapacity()); + } + + /** + * Checks that this resource is still valid and throws a MemoryInvalidException if it is not. + * Checks that the specified range of bytes is within bounds of this resource, throws + * {@link MemoryBoundsException} if it's not: i. e. if offsetBytes < 0, or length < 0, + * or offsetBytes + length > {@link #getCapacity()}. + * Checks that this operation is a read-only operation and throws a ReadOnlyException if not. + * @param offsetBytes the given offset in bytes of this object + * @param lengthBytes the given length in bytes of this object + * @Throws MemoryInvalidException if this resource is AutoCloseable and is no longer valid, i.e., + * it has already been closed. + * @Throws MemoryBoundsException if this resource violates the memory bounds of this resource. + * @Throws ReadOnlyException if the associated operation is not a Read-only operation. + */ + final void checkValidAndBoundsForWrite(final long offsetBytes, final long lengthBytes) { + checkValid(); + checkBounds(offsetBytes, lengthBytes, getCapacity()); + if (isReadOnly()) { + throw new ReadOnlyException("Memory is read-only."); + } + } + + @Override + public void close() { + /* Overridden by the leaf sub-classes that need AutoCloseable. */ + } + + @Override + public final boolean equalTo(final long thisOffsetBytes, final Resource that, + final long thatOffsetBytes, final long lengthBytes) { + if (that == null) { return false; } + return CompareAndCopy.equals(this, thisOffsetBytes, (ResourceImpl) that, thatOffsetBytes, lengthBytes); + } + + @Override + public void force() { //overridden by Map Leaves + throw new UnsupportedOperationException(NOT_MAPPED_FILE_RESOURCE); + } + + //Overridden by ByteBuffer Leaves. Used internally and for tests. + ByteBuffer getByteBuffer() { + return null; + } + + @Override + public final ByteOrder getTypeByteOrder() { + return isNativeOrder(getTypeId()) ? Util.NATIVE_BYTE_ORDER : Util.NON_NATIVE_BYTE_ORDER; + } + + @Override + public long getCapacity() { + checkValid(); + return capacityBytes; + } + + @Override + public long getCumulativeOffset(final long addOffsetBytes) { + return cumOffsetBytes + addOffsetBytes; + } + + @Override + public long getRelativeOffset() { + return offsetBytes; + } + + //Overridden by all leaves + int getTypeId() { + return typeId; + } + + //Overridden by Heap and ByteBuffer leaves. Made public as getArray() in BaseWritableMemoryImpl and BaseWritableBufferImpl + Object getUnsafeObject() { + return null; + } + + @Override + public boolean hasByteBuffer() { + return (getTypeId() & BYTEBUF) > 0; + } + + @Override + public final boolean isByteOrderCompatible(final ByteOrder byteOrder) { + final ByteOrder typeBO = getTypeByteOrder(); + return typeBO == ByteOrder.nativeOrder() && typeBO == byteOrder; + } + + static final boolean isBuffer(final int typeId) { + return (typeId & BUFFER) > 0; + } + + @Override + public boolean isCloseable() { + return (getTypeId() & (MAP | DIRECT)) > 0 && isAlive(); + } + + @Override + public final boolean isDirect() { + return getUnsafeObject() == null; + } + + @Override + public boolean isDuplicate() { + return (getTypeId() & DUPLICATE) > 0; + } + + @Override + public final boolean isHeap() { + checkValid(); + return getUnsafeObject() != null; + } + + @Override + public boolean isLoaded() { //overridden by Map Leaves + throw new IllegalStateException(NOT_MAPPED_FILE_RESOURCE); + } + + @Override + public boolean isMapped() { + return (getTypeId() & MAP) > 0; + } + + @Override + public boolean isMemory() { + return (getTypeId() & BUFFER) == 0; + } + + static final boolean isNativeOrder(final int typeId) { //not used + return (typeId & NONNATIVE_BO) == 0; + } + + @Override + public boolean isNonNativeOrder() { + return (getTypeId() & NONNATIVE_BO) > 0; + } + + @Override + public final boolean isReadOnly() { + checkValid(); + return (getTypeId() & READONLY) > 0; + } + + @Override + public boolean isRegionView() { + return (getTypeId() & REGION) > 0; + } + + @Override + public boolean isSameResource(final Resource that) { + checkValid(); + if (that == null) { return false; } + final ResourceImpl that1 = (ResourceImpl) that; + that1.checkValid(); + if (this == that1) { return true; } + return getCumulativeOffset(0) == that1.getCumulativeOffset(0) + && getCapacity() == that1.getCapacity() + && getUnsafeObject() == that1.getUnsafeObject() + && getByteBuffer() == that1.getByteBuffer(); + } + + //Overridden by Direct and Map leaves + @Override + public boolean isAlive() { + return true; + } + + @Override + public void load() { //overridden by Map leaves + throw new IllegalStateException(NOT_MAPPED_FILE_RESOURCE); + } + + private static String pad(final String s, final int fieldLen) { + return characterPad(s, fieldLen, ' ' , true); + } + + /** + * Returns first two number groups of the java version string. + * @param jdkVer the java version string from System.getProperty("java.version"). + * @return first two number groups of the java version string. + * @throws IllegalArgumentException for an improper Java version string. + */ + static int[] parseJavaVersion(final String jdkVer) { + final int p0, p1; + try { + String[] parts = jdkVer.trim().split("^0-9\\.");//grab only number groups and "." + parts = parts[0].split("\\."); //split out the number groups + p0 = Integer.parseInt(parts[0]); //the first number group + p1 = (parts.length > 1) ? Integer.parseInt(parts[1]) : 0; //2nd number group, or 0 + } catch (final NumberFormatException | ArrayIndexOutOfBoundsException e) { + throw new IllegalArgumentException("Improper Java -version string: " + jdkVer + LS + e); + } + checkJavaVersion(jdkVer, p0, p1); + return new int[] {p0, p1}; + } + + //REACHABILITY FENCE + static void reachabilityFence(final Object obj) { } + + final static int removeNnBuf(final int typeId) { return typeId & ~NONNATIVE_BO & ~BUFFER; } + + final static int setReadOnlyBit(final int typeId, final boolean readOnly) { + return readOnly ? typeId | READONLY : typeId & ~READONLY; + } + + /** + * Returns a formatted hex string of an area of this object. + * Used primarily for testing. + * @param state the ResourceImpl + * @param preamble a descriptive header + * @param offsetBytes offset bytes relative to the MemoryImpl start + * @param lengthBytes number of bytes to convert to a hex string + * @return a formatted hex string in a human readable array + */ + static final String toHex(final ResourceImpl state, final String preamble, final long offsetBytes, final int lengthBytes, + final boolean withData) { + final long capacity = state.getCapacity(); + ResourceImpl.checkBounds(offsetBytes, lengthBytes, capacity); + final StringBuilder sb = new StringBuilder(); + final Object uObj = state.getUnsafeObject(); + final String uObjStr; + final long uObjHeader; + if (uObj == null) { + uObjStr = "null"; + uObjHeader = 0; + } else { + uObjStr = uObj.getClass().getSimpleName() + ", " + (uObj.hashCode() & 0XFFFFFFFFL); + uObjHeader = UnsafeUtil.getArrayBaseOffset(uObj.getClass()); + } + final ByteBuffer bb = state.getByteBuffer(); + final String bbStr = bb == null ? "null" + : bb.getClass().getSimpleName() + ", " + (bb.hashCode() & 0XFFFFFFFFL); + final MemoryRequestServer memReqSvr = state.getMemoryRequestServer(); + final String memReqStr = memReqSvr != null + ? memReqSvr.getClass().getSimpleName() + ", " + (memReqSvr.hashCode() & 0XFFFFFFFFL) + : "null"; + final long cumBaseOffset = state.getCumulativeOffset(0); + sb.append(preamble).append(LS); + sb.append("UnsafeObj, hashCode : ").append(uObjStr).append(LS); + sb.append("UnsafeObjHeader : ").append(uObjHeader).append(LS); + sb.append("ByteBuf, hashCode : ").append(bbStr).append(LS); + sb.append("RegionOffset : ").append(state.getRelativeOffset()).append(LS); + if (ResourceImpl.isBuffer(state.typeId)) { + sb.append("Start : ").append(((PositionalImpl)state).getStart()).append(LS); + sb.append("Position : ").append(((PositionalImpl)state).getPosition()).append(LS); + sb.append("End : ").append(((PositionalImpl)state).getEnd()).append(LS); + } + sb.append("Capacity : ").append(capacity).append(LS); + sb.append("CumBaseOffset : ").append(cumBaseOffset).append(LS); + sb.append("MemReqSvr, hashCode : ").append(memReqStr).append(LS); + sb.append("is Alive : ").append(state.isAlive()).append(LS); + sb.append("Read Only : ").append(state.isReadOnly()).append(LS); + sb.append("Type Byte Order : ").append(state.getTypeByteOrder().toString()).append(LS); + sb.append("Native Byte Order : ").append(ByteOrder.nativeOrder().toString()).append(LS); + sb.append("JDK Runtime Version : ").append(JDK).append(LS); + //Data detail + if (withData) { + sb.append("Data, bytes : 0 1 2 3 4 5 6 7"); + + for (long i = 0; i < lengthBytes; i++) { + final int b = unsafe.getByte(uObj, cumBaseOffset + offsetBytes + i) & 0XFF; + if (i % 8 == 0) { //row header + sb.append(String.format("%n%20s: ", offsetBytes + i)); + } + sb.append(String.format("%02x ", b)); + } + sb.append(LS); + } + sb.append("### END SUMMARY"); + return sb.toString(); + } + + @Override + public final String toString(final String header, final long offsetBytes, final int lengthBytes, + final boolean withData) { + checkValid(); + final String klass = this.getClass().getSimpleName(); + final String s1 = String.format("(..., %d, %d)", offsetBytes, lengthBytes); + final long hcode = hashCode() & 0XFFFFFFFFL; + final String call = ".toHexString" + s1 + ", hashCode: " + hcode; + final StringBuilder sb = new StringBuilder(); + sb.append("### ").append(klass).append(" SUMMARY ###").append(LS); + sb.append("Type Info : ").append(typeDecode(typeId)).append(LS + LS); + sb.append("Header Comment : ").append(header).append(LS); + sb.append("Call Parameters : ").append(call); + return toHex(this, sb.toString(), offsetBytes, lengthBytes, withData); + } + + @Override + public final String toString() { + return toString("", 0, (int)this.getCapacity(), false); + } + + /** + * Decodes the resource type. This is primarily for debugging. + * @param typeId the given typeId + * @return a human readable string. + */ + static final String typeDecode(final int typeId) { + final StringBuilder sb = new StringBuilder(); + final int group1 = typeId & 0x7; + switch (group1) { // 0000 0XXX + case 0 : sb.append(pad("Writable + ",32)); break; + case 1 : sb.append(pad("ReadOnly + ",32)); break; + case 2 : sb.append(pad("Writable + Region + ",32)); break; + case 3 : sb.append(pad("ReadOnly + Region + ",32)); break; + case 4 : sb.append(pad("Writable + Duplicate + ",32)); break; + case 5 : sb.append(pad("ReadOnly + Duplicate + ",32)); break; + case 6 : sb.append(pad("Writable + Region + Duplicate + ",32)); break; + case 7 : sb.append(pad("ReadOnly + Region + Duplicate + ",32)); break; + default: break; + } + final int group2 = (typeId >>> 3) & 0x3; + switch (group2) { // 000X X000 + case 0 : sb.append(pad("Heap + ",15)); break; + case 1 : sb.append(pad("Direct + ",15)); break; + case 2 : sb.append(pad("Map + Direct + ",15)); break; + case 3 : sb.append(pad("Map + Direct + ",15)); break; + default: break; + } + final int group3 = (typeId >>> 5) & 0x1; + switch (group3) { // 00X0 0000 + case 0 : sb.append(pad("NativeOrder + ",17)); break; + case 1 : sb.append(pad("NonNativeOrder + ",17)); break; + default: break; + } + final int group4 = (typeId >>> 6) & 0x1; + switch (group4) { // 0X00 0000 + case 0 : sb.append(pad("Memory + ",9)); break; + case 1 : sb.append(pad("Buffer + ",9)); break; + default: break; + } + final int group5 = (typeId >>> 7) & 0x1; + switch (group5) { // X000 0000 + case 0 : sb.append(pad("",10)); break; + case 1 : sb.append(pad("ByteBuffer",10)); break; + default: break; + } + return sb.toString(); + } + + @Override + public final long xxHash64(final long offsetBytes, final long lengthBytes, final long seed) { + checkValid(); + return XxHash64.hash(getUnsafeObject(), getCumulativeOffset(0) + offsetBytes, lengthBytes, seed); + } + + @Override + public final long xxHash64(final long in, final long seed) { + return XxHash64.hash(in, seed); + } + +} From a3209b0b7262664a9f9e496d8a7ef98b43b58f87 Mon Sep 17 00:00:00 2001 From: Ruifeng Zheng Date: Mon, 11 May 2026 19:30:27 +0800 Subject: [PATCH 094/286] [MINOR][INFRA][4.2] Fix `branch` input default in build_and_test.yml ### What changes were proposed in this pull request? Update the default value of the `branch` workflow input in `.github/workflows/build_and_test.yml` from `master` to `branch-4.2`, and drop the stale `# Change 'master' to 'branch-4.0' in branch-4.0 branch after cutting it.` comment. ### Why are the changes needed? When `branch-4.2` was cut from master, the comment in `build_and_test.yml` flagged that the default should be updated, but it wasn't. As a result, the `Build` workflow on pushes to `branch-4.2` invokes the reusable `build_and_test.yml` without an explicit `branch` input, which falls back to the default `master`. The `precondition` job then runs `actions/checkout` with `ref: master` against `apache/spark`, so the CI tests `master` rather than `branch-4.2`. For example, [this `branch-4.2` push CI run](https://github.com/apache/spark/actions/runs/25591666487/job/75130390722) tested `master` instead of `branch-4.2`. The same fix was already applied on `branch-4.0` and `branch-4.1`. ### Does this PR introduce _any_ user-facing change? No. CI-only. ### How was this patch tested? GitHub Actions on this PR. The default takes effect on subsequent pushes to `branch-4.2`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code Closes #55796 from zhengruifeng/fix-branch-default-4.2. Authored-by: Ruifeng Zheng Signed-off-by: Ruifeng Zheng --- .github/workflows/build_and_test.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index ffcc49eaec1bf..99223efbf0dd2 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -30,8 +30,7 @@ on: description: Branch to run the build against required: false type: string - # Change 'master' to 'branch-4.0' in branch-4.0 branch after cutting it. - default: master + default: branch-4.2 hadoop: description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it. required: false From 9cc4a77f4e5a340c16a97d2c1ba6a119f32c8d82 Mon Sep 17 00:00:00 2001 From: YangJie Date: Mon, 11 May 2026 20:16:30 +0800 Subject: [PATCH 095/286] [SPARK-56586][CONNECT][TESTS] Bound and retry the flaky python foreachBatch termination test ### What changes were proposed in this pull request? Harden `python foreachBatch process: process terminates after query is stopped` in `SparkConnectSessionHolderSuite`: - Bound `query.stop()` with `spark.sql.streaming.stopTimeout = 30s` so a stuck streaming batch cannot wait forever. - Run each attempt on a daemon thread capped at 2 minutes. On timeout, close the Python worker sockets via `cleanerCache.cleanUpAll()` to unblock the `dataIn.readInt` hang, interrupt the worker, and allow a 30s grace period for its own `finally` to run. - Retry up to 3 times. Retry notices go to stdout so they appear in the GitHub Actions job log; `SparkFunSuite.retry` uses log4j, which only writes to `target/unit-tests.log`. - Scope cleanup per attempt so a leaked worker from a timed-out attempt does not interfere with the next one: unique query names, identity-checked `SparkConnectService.stop()`, and listener removal restricted to listeners this attempt registered. - Wrap every cleanup step in a `runQuietly` helper so a failure there cannot mask a primary test failure. ### Why are the changes needed? Without these bounds, a hang in the Python foreachBatch worker's non-interruptible `dataIn.readInt` leaves the test thread running indefinitely; the failing CI run sat there for ~150 minutes before the outer job timeout killed it. With the bounds and retry, a single stuck attempt recovers instead of burning the CI slot. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` build/sbt 'connect/testOnly *SparkConnectSessionHolderSuite -- -z "python foreachBatch"' ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55786 from LuciferYang/fix-foreach-batch-flaky-test. Lead-authored-by: YangJie Co-authored-by: Ruifeng Zheng Signed-off-by: yangjie01 (cherry picked from commit 59b4bbb2664d9088da5b5745419cb8ddd61fb3fb) Signed-off-by: yangjie01 --- .../SparkConnectSessionHolderSuite.scala | 184 ++++++++++++++++-- 1 file changed, 169 insertions(+), 15 deletions(-) diff --git a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala index 17402ab5ddb43..cff5f345d2573 100644 --- a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala +++ b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala @@ -19,11 +19,13 @@ package org.apache.spark.sql.connect.service import java.nio.charset.StandardCharsets import java.nio.file.Files +import java.util.concurrent.{TimeoutException, TimeUnit} import scala.collection.mutable import scala.jdk.CollectionConverters._ import scala.sys.process.Process import scala.util.Random +import scala.util.control.NonFatal import com.google.common.collect.Lists import org.scalatest.time.SpanSugar._ @@ -37,8 +39,10 @@ import org.apache.spark.sql.connect.common.InvalidPlanInput import org.apache.spark.sql.connect.config.Connect import org.apache.spark.sql.connect.planner.{PythonStreamingQueryListener, SparkConnectPlanner, StreamingForeachBatchHelper} import org.apache.spark.sql.connect.planner.StreamingForeachBatchHelper.RunnerCleaner +import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.pipelines.graph.{DataflowGraph, PipelineUpdateContextImpl} import org.apache.spark.sql.pipelines.logging.PipelineEvent +import org.apache.spark.sql.streaming.StreamingQueryListener import org.apache.spark.sql.test.SharedSparkSession import org.apache.spark.util.ArrayImplicits._ @@ -228,15 +232,117 @@ class SparkConnectSessionHolderSuite extends SharedSparkSession { } } - test("python foreachBatch process: process terminates after query is stopped") { - // scalastyle:off assume - assume(IntegratedUDFTestUtils.shouldTestPandasUDFs) - assume(PythonTestDepsChecker.isConnectDepsAvailable) - // scalastyle:on assume + // Log and swallow best-effort cleanup failures so they do not mask a primary test + // failure. InterruptedException re-asserts the interrupt flag on the current thread; + // fatal errors (OOM, StackOverflow, LinkageError) propagate. + private def runQuietly(label: String, op: => Unit): Unit = { + try op + catch { + case _: InterruptedException => Thread.currentThread().interrupt() + case NonFatal(t) => + // scalastyle:off println + println(s"===== $label suppressed ${t.getClass.getSimpleName}: ${t.getMessage} =====") + // scalastyle:on println + } + } + + // Same semantics as SparkFunSuite.retry, but prints to stdout so retries show up in the + // GitHub Actions job log (SparkFunSuite.retry's log4j output only lands in + // target/unit-tests.log, surfaced as an artifact rather than in the live log). + private def retryWithVisibleLog(maxAttempts: Int)(body: => Unit): Unit = { + var attempt = 1 + var done = false + while (!done) { + try { + body + done = true + } catch { + case NonFatal(t) if attempt >= maxAttempts => throw t + case NonFatal(t) => + // scalastyle:off println + println( + s"===== Attempt $attempt/$maxAttempts failed " + + s"(${t.getClass.getSimpleName}: ${t.getMessage}); retrying =====") + // scalastyle:on println + // A leaked worker from this attempt may still hold sockets/listeners; do not + // let afterEach/beforeEach throwing on that residual state abort the retry loop. + runQuietly("afterEach", afterEach()) + runQuietly("beforeEach", beforeEach()) + attempt += 1 + } + } + } + + private def awaitTestBodyInNewThread(timeoutMillis: Long, onTimeout: () => Unit)( + body: => Unit): Unit = { + @volatile var error: Throwable = null + val runnable: Runnable = () => { + try { + body + } catch { + case t: Throwable => error = t + } + } + val worker = new Thread(runnable, s"${getClass.getSimpleName}-testBody-worker") + worker.setDaemon(true) + worker.start() + worker.join(timeoutMillis) + if (worker.isAlive) { + // Capture the worker's stack so post-mortem diagnostics can identify which leaked + // thread belongs to which attempt without a separate jstack. + // scalastyle:off println + println( + s"===== Test body did not complete within $timeoutMillis ms " + + s"(thread=${worker.getName}, state=${worker.getState}); stack trace follows =====") + worker.getStackTrace.foreach(frame => println(s" at $frame")) + // scalastyle:on println + // Best-effort: release any resource the worker is blocked on so it can unwind its own + // finally and stop holding global state (SparkConnectService, listeners, ...). + onTimeout() + // Also interrupt the worker so any interruptible blocking call (e.g. the Thread.join + // inside StreamExecution.interruptAndAwaitExecutionThreadTermination) wakes up. + worker.interrupt() + // Grace period for the now-unblocked worker to run its own finally + // (SparkConnectService.stop() then the ~4s settle sleep). + val gracePeriodMs = 30.seconds.toMillis + worker.join(gracePeriodMs) + val te = new TimeoutException( + s"Test body did not complete within $timeoutMillis ms " + + s"(after a $gracePeriodMs ms post-cleanup grace period)") + // If the body finished during the grace window, surface the original failure + // as the cause so a slow assertion failure is not misreported as a pure hang. + if (!worker.isAlive && error != null) te.initCause(error) + throw te + } + if (error != null) throw error + } + + private def runPythonForeachBatchTerminationTestBody(sessionHolder: SessionHolder): Unit = { + // Unique query names per attempt: a leaked query from a timed-out attempt may still + // occupy the old name in spark.streams.active. + val suffix = s"_${System.nanoTime()}" + val q1Name = s"foreachBatch_termination_test_q1$suffix" + val q2Name = s"foreachBatch_termination_test_q2$suffix" + + // Snapshot listeners before this attempt registers anything so we can scope cleanup and + // assertions to listeners we added -- even if a previous timed-out attempt leaked a worker + // whose own finally is racing with us. + val baselineListeners = spark.streams.listListeners().toSet + var capturedServer: AnyRef = null + var ourNewListeners = Set.empty[StreamingQueryListener] - val sessionHolder = SparkConnectTestUtils.createDummySessionHolder(spark) try { + // A previous timed-out attempt's leaked worker may still hold `started=true`, which + // would make `start()` below a no-op and cause this attempt to share (and later + // re-stop) the stale server. Force-stop first so `start()` creates a fresh instance; + // the identity check in `finally` then distinguishes attempts. + if (SparkConnectService.started) { + runQuietly("stale SparkConnectService.stop()", SparkConnectService.stop()) + } SparkConnectService.start(spark.sparkContext) + // Identity-check the server in `finally`: a previous attempt's leaked finally must + // not tear down a service belonging to a later attempt. + capturedServer = SparkConnectService.server val pythonFn = dummyPythonFunction(sessionHolder)(streamingForeachBatchFunction) val (fn1, cleaner1) = @@ -249,7 +355,7 @@ class SparkConnectSessionHolderSuite extends SharedSparkSession { .load() .writeStream .format("memory") - .queryName("foreachBatch_termination_test_q1") + .queryName(q1Name) .foreachBatch(fn1) .start() @@ -258,7 +364,7 @@ class SparkConnectSessionHolderSuite extends SharedSparkSession { .load() .writeStream .format("memory") - .queryName("foreachBatch_termination_test_q2") + .queryName(q2Name) .foreachBatch(fn2) .start() @@ -267,6 +373,10 @@ class SparkConnectSessionHolderSuite extends SharedSparkSession { sessionHolder.streamingForeachBatchRunnerCleanerCache .registerCleanerForQuery(query2, cleaner2) + // The first registerCleanerForQuery lazily registers the cleaner listener. Capture the + // listeners we added so finally only removes ours, not a concurrent attempt's. + ourNewListeners = spark.streams.listListeners().toSet -- baselineListeners + val (runner1, runner2) = (cleaner1.asInstanceOf[RunnerCleaner].runner, cleaner2.asInstanceOf[RunnerCleaner].runner) @@ -288,14 +398,58 @@ class SparkConnectSessionHolderSuite extends SharedSparkSession { assert(runner2.isWorkerStopped().get) } - assert(spark.streams.active.isEmpty) // no running query - assert(spark.streams.listListeners().length == 1) // only process termination listener + // Only assert this attempt's queries stopped; a previous timed-out attempt may have + // leaked queries into spark.streams.active that we cannot synchronously clean up. + assert(!spark.streams.active.exists(q => q.name == q1Name || q.name == q2Name)) + // Scoped to this attempt: exactly one new listener (the cleaner listener) should + // have been registered, regardless of any listeners leaked by a prior attempt. + assert( + ourNewListeners.size == 1, + s"expected exactly 1 new listener registered by this attempt, " + + s"got ${ourNewListeners.size}") } finally { - SparkConnectService.stop() - // Wait for things to calm down. - Thread.sleep(4.seconds.toMillis) - // remove process termination listener - spark.streams.listListeners().foreach(spark.streams.removeListener) + // Only stop the service if it is still the one this attempt started; otherwise a + // previous attempt's leaked finally would tear down the live service of the current + // attempt. + if (capturedServer != null && (SparkConnectService.server eq capturedServer)) { + // Cleanup is best-effort: any failure must not mask the primary failure in the + // try block, and the listener cleanup below must still run. + runQuietly("SparkConnectService.stop()", SparkConnectService.stop()) + runQuietly("settle sleep", Thread.sleep(4.seconds.toMillis)) + } + // Remove only the listeners this attempt registered; never touch a concurrent + // attempt's process-termination listener. Wrapped in `runQuietly` so a throw here + // cannot mask a primary failure in the try block. + runQuietly("removeListeners", ourNewListeners.foreach(spark.streams.removeListener)) + } + } + + test("python foreachBatch process: process terminates after query is stopped") { + // scalastyle:off assume + assume(IntegratedUDFTestUtils.shouldTestPandasUDFs) + assume(PythonTestDepsChecker.isConnectDepsAvailable) + // scalastyle:on assume + + // Bound query.stop() so it cannot hang indefinitely: spark.sql.streaming.stopTimeout + // defaults to 0 (wait forever), which turns a stuck batch into an unkillable test. + // 30s is small enough to fit under the outer per-attempt cap with room to spare. + withSQLConf(SQLConf.STREAMING_STOP_TIMEOUT.key -> "30000") { + retryWithVisibleLog(maxAttempts = 3) { + // Run the body on a fresh daemon thread so the test thread can recover from a + // hang in a non-interruptible socket read. SessionHolder is created outside the + // body so onTimeout can close its Python worker sockets via cleanerCache; that + // unblocks the hung dataIn.readInt so the leaked thread's finally can settle + // before the next retry. 2-minute cap strictly bounds the original 150-minute hang. + val sessionHolder = SparkConnectTestUtils.createDummySessionHolder(spark) + awaitTestBodyInNewThread( + timeoutMillis = TimeUnit.MINUTES.toMillis(2), + onTimeout = () => + runQuietly( + "onTimeout cleanUpAll", + sessionHolder.streamingForeachBatchRunnerCleanerCache.cleanUpAll())) { + runPythonForeachBatchTerminationTestBody(sessionHolder) + } + } } } From 78c9adf2703b7a12e3c9042e7a8a87a101821290 Mon Sep 17 00:00:00 2001 From: Mihailo Timotic Date: Mon, 11 May 2026 20:39:34 +0800 Subject: [PATCH 096/286] [SPARK-56814][SQL][TESTS] Add lateral join tests for outer attribute visibility after NATURAL/USING JOIN ## What changes were proposed in this pull request? Add SQL query test cases to join-lateral.sql covering lateral join outer attribute visibility after NATURAL JOIN and USING JOIN. Specifically: 1. Lateral after NATURAL JOIN -- unqualified key: verifies the merged join key resolves correctly in the lateral subquery. 2. Lateral after NATURAL JOIN -- qualified keys: verifies t1.k and t2.k resolve to the original pre-merge columns. 3. Lateral after USING JOIN -- qualified keys: same as above but with explicit USING (k) syntax. 4. Lateral cannot see column hidden by subquery alias: verifies that a column not in the subquery's output (v1 behind SELECT k FROM ... ORDER BY v1) is not visible to the lateral subquery. 5. Lateral cannot see column not in GROUP BY output: verifies that a column dropped by GROUP BY projection (v1 in SELECT k FROM ... GROUP BY k) is not visible to the lateral subquery. ## Why are the changes needed? The existing join-lateral.sql tests cover NATURAL JOIN and USING JOIN with the lateral subquery itself (lines 28-29), but do not test lateral joins chained after a NATURAL/USING JOIN -- i.e., whether the merged/qualified/hidden columns from the left side are correctly visible or hidden across the lateral boundary. These tests close that gap. ## Does this PR introduce any user-facing change? No. ## How was this patch tested? Added test cases ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-4-6) Closes #55794 from mihailotim-db/mihailo-timotic_data/lateral-join-outer-attr-visibility. Authored-by: Mihailo Timotic Signed-off-by: Wenchen Fan (cherry picked from commit 52daa61338998b915b51e2fd221a82087beea26d) Signed-off-by: Wenchen Fan --- .../analyzer-results/join-lateral.sql.out | 130 ++++++++++++++++++ .../sql-tests/inputs/join-lateral.sql | 30 ++++ .../sql-tests/results/join-lateral.sql.out | 81 +++++++++++ 3 files changed, 241 insertions(+) diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/join-lateral.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/join-lateral.sql.out index 4666e62b2d690..cb46c265512ff 100644 --- a/sql/core/src/test/resources/sql-tests/analyzer-results/join-lateral.sql.out +++ b/sql/core/src/test/resources/sql-tests/analyzer-results/join-lateral.sql.out @@ -3064,6 +3064,136 @@ Project [1 AS 1#x] +- LocalRelation [col1#x, col2#x] +-- !query +WITH nj1(k, v1) AS (VALUES (1, 'a')), + nj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM nj1 NATURAL JOIN nj2, +LATERAL (SELECT k AS unq_k) +-- !query analysis +WithCTE +:- CTERelationDef xxxx, false +: +- SubqueryAlias nj1 +: +- Project [col1#x AS k#x, col2#x AS v1#x] +: +- LocalRelation [col1#x, col2#x] +:- CTERelationDef xxxx, false +: +- SubqueryAlias nj2 +: +- Project [col1#x AS k#x, col2#x AS v2#x] +: +- LocalRelation [col1#x, col2#x] ++- Project [k#x, v1#x, v2#x, unq_k#x] + +- LateralJoin lateral-subquery#x [k#x], Inner + : +- SubqueryAlias __auto_generated_subquery_name + : +- Project [outer(k#x) AS unq_k#x] + : +- OneRowRelation + +- Project [k#x, v1#x, v2#x] + +- Join Inner, (k#x = k#x) + :- SubqueryAlias nj1 + : +- CTERelationRef xxxx, true, [k#x, v1#x], false, false, 1 + +- SubqueryAlias nj2 + +- CTERelationRef xxxx, true, [k#x, v2#x], false, false, 1 + + +-- !query +WITH nj1(k, v1) AS (VALUES (1, 'a')), + nj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM nj1 NATURAL JOIN nj2, +LATERAL (SELECT k AS unq_k, nj1.k AS qual_nj1k, nj2.k AS qual_nj2k) +-- !query analysis +WithCTE +:- CTERelationDef xxxx, false +: +- SubqueryAlias nj1 +: +- Project [col1#x AS k#x, col2#x AS v1#x] +: +- LocalRelation [col1#x, col2#x] +:- CTERelationDef xxxx, false +: +- SubqueryAlias nj2 +: +- Project [col1#x AS k#x, col2#x AS v2#x] +: +- LocalRelation [col1#x, col2#x] ++- Project [k#x, v1#x, v2#x, unq_k#x, qual_nj1k#x, qual_nj2k#x] + +- Project [k#x, v1#x, v2#x, unq_k#x, qual_nj1k#x, qual_nj2k#x] + +- LateralJoin lateral-subquery#x [k#x && k#x && k#x], Inner + : +- SubqueryAlias __auto_generated_subquery_name + : +- Project [outer(k#x) AS unq_k#x, outer(k#x) AS qual_nj1k#x, outer(k#x) AS qual_nj2k#x] + : +- OneRowRelation + +- Project [k#x, v1#x, v2#x, k#x] + +- Join Inner, (k#x = k#x) + :- SubqueryAlias nj1 + : +- CTERelationRef xxxx, true, [k#x, v1#x], false, false, 1 + +- SubqueryAlias nj2 + +- CTERelationRef xxxx, true, [k#x, v2#x], false, false, 1 + + +-- !query +WITH uj1(k, v1) AS (VALUES (1, 'a')), + uj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM uj1 JOIN uj2 USING (k), +LATERAL (SELECT k AS unq_k, uj1.k AS qual_uj1k, uj2.k AS qual_uj2k) +-- !query analysis +WithCTE +:- CTERelationDef xxxx, false +: +- SubqueryAlias uj1 +: +- Project [col1#x AS k#x, col2#x AS v1#x] +: +- LocalRelation [col1#x, col2#x] +:- CTERelationDef xxxx, false +: +- SubqueryAlias uj2 +: +- Project [col1#x AS k#x, col2#x AS v2#x] +: +- LocalRelation [col1#x, col2#x] ++- Project [k#x, v1#x, v2#x, unq_k#x, qual_uj1k#x, qual_uj2k#x] + +- Project [k#x, v1#x, v2#x, unq_k#x, qual_uj1k#x, qual_uj2k#x] + +- LateralJoin lateral-subquery#x [k#x && k#x && k#x], Inner + : +- SubqueryAlias __auto_generated_subquery_name + : +- Project [outer(k#x) AS unq_k#x, outer(k#x) AS qual_uj1k#x, outer(k#x) AS qual_uj2k#x] + : +- OneRowRelation + +- Project [k#x, v1#x, v2#x, k#x] + +- Join Inner, (k#x = k#x) + :- SubqueryAlias uj1 + : +- CTERelationRef xxxx, true, [k#x, v1#x], false, false, 1 + +- SubqueryAlias uj2 + +- CTERelationRef xxxx, true, [k#x, v2#x], false, false, 1 + + +-- !query +WITH cte1(k, v1) AS (VALUES (1, 'a')) +SELECT * FROM (SELECT k FROM cte1 ORDER BY v1) sub, +LATERAL (SELECT v1 AS leaked) +-- !query analysis +org.apache.spark.sql.catalyst.ExtendedAnalysisException +{ + "errorClass" : "UNRESOLVED_COLUMN.WITHOUT_SUGGESTION", + "sqlState" : "42703", + "messageParameters" : { + "objectName" : "`v1`" + }, + "queryContext" : [ { + "objectType" : "", + "objectName" : "", + "startIndex" : 107, + "stopIndex" : 108, + "fragment" : "v1" + } ] +} + + +-- !query +WITH cte1(k, v1) AS (VALUES (1, 'a'), (2, 'b'), (3, 'c')) +SELECT * FROM (SELECT k FROM cte1 GROUP BY k) g, +LATERAL (SELECT v1 AS leaked) +-- !query analysis +org.apache.spark.sql.catalyst.ExtendedAnalysisException +{ + "errorClass" : "UNRESOLVED_COLUMN.WITHOUT_SUGGESTION", + "sqlState" : "42703", + "messageParameters" : { + "objectName" : "`v1`" + }, + "queryContext" : [ { + "objectType" : "", + "objectName" : "", + "startIndex" : 124, + "stopIndex" : 125, + "fragment" : "v1" + } ] +} + + -- !query DROP VIEW t1 -- !query analysis diff --git a/sql/core/src/test/resources/sql-tests/inputs/join-lateral.sql b/sql/core/src/test/resources/sql-tests/inputs/join-lateral.sql index e3cef9207d20f..8a71afb38a76f 100644 --- a/sql/core/src/test/resources/sql-tests/inputs/join-lateral.sql +++ b/sql/core/src/test/resources/sql-tests/inputs/join-lateral.sql @@ -552,6 +552,36 @@ left join order by t_inner.b1,t_inner.b2 desc limit 1 ) as lateral_table; +-- lateral join after NATURAL/USING JOIN: outer attribute visibility + +-- lateral after NATURAL JOIN: unqualified key resolves to the merged column +WITH nj1(k, v1) AS (VALUES (1, 'a')), + nj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM nj1 NATURAL JOIN nj2, +LATERAL (SELECT k AS unq_k); + +-- lateral after NATURAL JOIN: qualified keys resolve to original columns +WITH nj1(k, v1) AS (VALUES (1, 'a')), + nj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM nj1 NATURAL JOIN nj2, +LATERAL (SELECT k AS unq_k, nj1.k AS qual_nj1k, nj2.k AS qual_nj2k); + +-- lateral after USING JOIN: unqualified and qualified keys +WITH uj1(k, v1) AS (VALUES (1, 'a')), + uj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM uj1 JOIN uj2 USING (k), +LATERAL (SELECT k AS unq_k, uj1.k AS qual_uj1k, uj2.k AS qual_uj2k); + +-- lateral cannot see column hidden by a subquery alias +WITH cte1(k, v1) AS (VALUES (1, 'a')) +SELECT * FROM (SELECT k FROM cte1 ORDER BY v1) sub, +LATERAL (SELECT v1 AS leaked); + +-- lateral cannot see column not in GROUP BY output +WITH cte1(k, v1) AS (VALUES (1, 'a'), (2, 'b'), (3, 'c')) +SELECT * FROM (SELECT k FROM cte1 GROUP BY k) g, +LATERAL (SELECT v1 AS leaked); + -- clean up DROP VIEW t1; DROP VIEW t2; diff --git a/sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out b/sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out index 11bafb2cf63c9..b8af8dfea2211 100644 --- a/sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/join-lateral.sql.out @@ -1905,6 +1905,87 @@ struct<1:int> 1 +-- !query +WITH nj1(k, v1) AS (VALUES (1, 'a')), + nj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM nj1 NATURAL JOIN nj2, +LATERAL (SELECT k AS unq_k) +-- !query schema +struct +-- !query output +1 a b 1 + + +-- !query +WITH nj1(k, v1) AS (VALUES (1, 'a')), + nj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM nj1 NATURAL JOIN nj2, +LATERAL (SELECT k AS unq_k, nj1.k AS qual_nj1k, nj2.k AS qual_nj2k) +-- !query schema +struct +-- !query output +1 a b 1 1 1 + + +-- !query +WITH uj1(k, v1) AS (VALUES (1, 'a')), + uj2(k, v2) AS (VALUES (1, 'b')) +SELECT * FROM uj1 JOIN uj2 USING (k), +LATERAL (SELECT k AS unq_k, uj1.k AS qual_uj1k, uj2.k AS qual_uj2k) +-- !query schema +struct +-- !query output +1 a b 1 1 1 + + +-- !query +WITH cte1(k, v1) AS (VALUES (1, 'a')) +SELECT * FROM (SELECT k FROM cte1 ORDER BY v1) sub, +LATERAL (SELECT v1 AS leaked) +-- !query schema +struct<> +-- !query output +org.apache.spark.sql.catalyst.ExtendedAnalysisException +{ + "errorClass" : "UNRESOLVED_COLUMN.WITHOUT_SUGGESTION", + "sqlState" : "42703", + "messageParameters" : { + "objectName" : "`v1`" + }, + "queryContext" : [ { + "objectType" : "", + "objectName" : "", + "startIndex" : 107, + "stopIndex" : 108, + "fragment" : "v1" + } ] +} + + +-- !query +WITH cte1(k, v1) AS (VALUES (1, 'a'), (2, 'b'), (3, 'c')) +SELECT * FROM (SELECT k FROM cte1 GROUP BY k) g, +LATERAL (SELECT v1 AS leaked) +-- !query schema +struct<> +-- !query output +org.apache.spark.sql.catalyst.ExtendedAnalysisException +{ + "errorClass" : "UNRESOLVED_COLUMN.WITHOUT_SUGGESTION", + "sqlState" : "42703", + "messageParameters" : { + "objectName" : "`v1`" + }, + "queryContext" : [ { + "objectType" : "", + "objectName" : "", + "startIndex" : 124, + "stopIndex" : 125, + "fragment" : "v1" + } ] +} + + -- !query DROP VIEW t1 -- !query schema From 36cc5f64007529261d437cd53bb915f8465c3e14 Mon Sep 17 00:00:00 2001 From: Cheng Pan Date: Mon, 11 May 2026 21:08:03 +0800 Subject: [PATCH 097/286] [SPARK-56815][SPARK-55852][DOCS] Document Java 25 support ### What changes were proposed in this pull request? Update docs to mention JDK 25 support, in addition, mark JDK 25 prior to 25.0.3 as deprecated due to the data corruption report with G1GC mentioned in SPARK-55852. ### Why are the changes needed? Keep docs up-to-date. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review, because it's a doc-only change. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55798 from pan3793/SPARK-56815. Authored-by: Cheng Pan Signed-off-by: Cheng Pan (cherry picked from commit 92a71c6e3d872ef4ec4c13c85b6c341a5f28af84) Signed-off-by: Cheng Pan --- docs/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index cb32ddcde7e2b..6d590172e9380 100644 --- a/docs/index.md +++ b/docs/index.md @@ -34,7 +34,8 @@ source, visit [Building Spark](building-spark.html). Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS), and it should run on any platform that runs a supported version of Java. This should include JVMs on x86_64 and ARM64. It's easy to run locally on one machine --- all you need is to have `java` installed on your system `PATH`, or the `JAVA_HOME` environment variable pointing to a Java installation. -Spark runs on Java 17/21, Scala 2.13, Python 3.10+, and R 3.5+ (Deprecated). +Spark runs on Java 17/21/25, Scala 2.13, Python 3.10+, and R 3.5+ (Deprecated). +Java 25 prior to version 25.0.3 support is deprecated as of Spark 4.2.0. When using the Scala API, it is necessary for applications to use the same version of Scala that Spark was compiled for. Since Spark 4.0.0, it's Scala 2.13. # Running the Examples and Shell From f34a0148986f83870e2e727666344ac0a795ec10 Mon Sep 17 00:00:00 2001 From: jameswillis Date: Mon, 11 May 2026 10:23:10 -0700 Subject: [PATCH 098/286] [SPARK-55897][SQL] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? `ColumnarRow.get()`, `ColumnarBatchRow.get()`, and `ColumnarArray.get()` throw `SparkUnsupportedOperationException` when called with a `UserDefinedType` because they have no branch to handle UDTs. This PR adds UDT handling to all three methods: - **ColumnarRow** and **ColumnarBatchRow**: Add an `instanceof UserDefinedType` branch that recurses with `udt.sqlType()`, matching the pattern already used in `SpecializedGettersReader.read()`. - **ColumnarArray**: Change the `handleUserDefinedType` flag from `false` to `true` in the existing call to `SpecializedGettersReader.read()`. ### Why are the changes needed? The codegen path (`CodeGenerator.getValue()`) unwraps `udt.sqlType()` before generating accessor calls, so UDT columns work when whole-stage codegen is active. However, on the interpreted eval path — when codegen is disabled, falls back, or the number of fields exceeds `spark.sql.codegen.maxFields` — `GetStructField.nullSafeEval` calls `ColumnarRow.get(ordinal, udtType)` directly, which hits the unhandled branch and throws. ### Does this PR introduce _any_ user-facing change? Yes. UDT columns in columnar data sources (e.g., Parquet) now work correctly on the interpreted evaluation path. Previously they would throw `SparkUnsupportedOperationException`. ### How was this patch tested? Added 6 new tests in `ColumnarBatchSuite` covering all 3 methods × 2 UDT backing types (primitive `IntegerType` and complex `StructType`). Each test creates columnar vectors with UDT data and verifies that `get()` returns the correct value. Two helper UDT classes (`TestIntUDT`, `TestStructWrapperUDT`) are defined for the tests. ### Was this patch authored or co-authored using generative AI tooling? Yes. Opus 4.6 Closes #54701 from james-willis/columnar-row-udt-test. Authored-by: jameswillis Signed-off-by: Huaxin Gao (cherry picked from commit 472735cefef51159b0ab332fa12705c67e146d42) Signed-off-by: Huaxin Gao --- .../spark/sql/vectorized/ColumnarArray.java | 2 +- .../sql/vectorized/ColumnarBatchRow.java | 2 + .../spark/sql/vectorized/ColumnarRow.java | 2 + .../vectorized/ColumnarBatchSuite.scala | 121 ++++++++++++++++++ 4 files changed, 126 insertions(+), 1 deletion(-) diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java index fad1817aca199..861a6a4c50e44 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java @@ -213,7 +213,7 @@ public ColumnarMap getMap(int ordinal) { @Override public Object get(int ordinal, DataType dataType) { - return SpecializedGettersReader.read(this, ordinal, dataType, false, false); + return SpecializedGettersReader.read(this, ordinal, dataType, false, true); } @Override diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java index 3d1e780f6e057..42b335dfd2bc1 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java @@ -215,6 +215,8 @@ public Object get(int ordinal, DataType dataType) { return getMap(ordinal); } else if (dataType instanceof VariantType) { return getVariant(ordinal); + } else if (dataType instanceof UserDefinedType udt) { + return get(ordinal, udt.sqlType()); } else { throw new SparkUnsupportedOperationException( "_LEGACY_ERROR_TEMP_3152", Map.of("dataType", String.valueOf(dataType))); diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java index 656c5f8a8f30e..d66baa8fd8fe3 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java @@ -217,6 +217,8 @@ public Object get(int ordinal, DataType dataType) { return getMap(ordinal); } else if (dataType instanceof VariantType) { return getVariant(ordinal); + } else if (dataType instanceof UserDefinedType udt) { + return get(ordinal, udt.sqlType()); } else { throw new SparkUnsupportedOperationException("_LEGACY_ERROR_TEMP_3155"); } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala index 0f2ca93f287c7..40f73450eb21d 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala @@ -48,6 +48,38 @@ import org.apache.spark.unsafe.Platform import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String, VariantVal} import org.apache.spark.util.ArrayImplicits._ +/** + * A minimal UDT backed by IntegerType, used by SPARK-55897 tests. + */ +@SQLUserDefinedType(udt = classOf[TestIntUDT]) +private case class TestIntWrapper(value: Int) + +private class TestIntUDT extends UserDefinedType[TestIntWrapper] { + override def sqlType: DataType = IntegerType + override def serialize(obj: TestIntWrapper): Any = obj.value + override def userClass: Class[TestIntWrapper] = classOf[TestIntWrapper] + override def deserialize(datum: Any): TestIntWrapper = datum match { + case v: Int => TestIntWrapper(v) + } +} + +/** + * A minimal UDT backed by StructType, used by SPARK-55897 tests. + */ +@SQLUserDefinedType(udt = classOf[TestStructWrapperUDT]) +private case class TestStructWrapper(x: Int, y: Long) + +private class TestStructWrapperUDT extends UserDefinedType[TestStructWrapper] { + override def sqlType: DataType = new StructType() + .add("x", IntegerType) + .add("y", LongType) + override def serialize(obj: TestStructWrapper): Any = InternalRow(obj.x, obj.y) + override def userClass: Class[TestStructWrapper] = classOf[TestStructWrapper] + override def deserialize(datum: Any): TestStructWrapper = datum match { + case row: InternalRow => TestStructWrapper(row.getInt(0), row.getLong(1)) + } +} + @ExtendedSQLTest class ColumnarBatchSuite extends SparkFunSuite { @@ -2071,4 +2103,93 @@ class ColumnarBatchSuite extends SparkFunSuite { } } } + + testVector( + "SPARK-55897: ColumnarRow.get with primitive-backed UDT", + 10, + new StructType().add("name", StringType).add("udt_field", IntegerType)) { column => + column.getChild(0).putByteArray(0, "hello".getBytes) + column.getChild(1).putInt(0, 42) + + val row = column.getStruct(0) + assert(row.get(1, new TestIntUDT()) === 42) + } + + testVector( + "SPARK-55897: ColumnarRow.get with struct-backed UDT", + 10, + new StructType() + .add("id", IntegerType) + .add("nested", new StructType().add("x", IntegerType).add("y", LongType))) { column => + column.getChild(0).putInt(0, 1) + column.getChild(1).getChild(0).putInt(0, 10) + column.getChild(1).getChild(1).putLong(0, 20L) + + val row = column.getStruct(0) + val nested = row.get(1, new TestStructWrapperUDT()).asInstanceOf[InternalRow] + assert(nested.getInt(0) === 10) + assert(nested.getLong(1) === 20L) + } + + testVector( + "SPARK-55897: ColumnarArray.get with primitive-backed UDT", + 10, + new ArrayType(IntegerType, false)) { column => + val data = column.arrayData() + data.putInt(0, 10) + data.putInt(1, 20) + column.putArray(0, 0, 2) + + val arr = column.getArray(0) + assert(arr.get(0, new TestIntUDT()) === 10) + assert(arr.get(1, new TestIntUDT()) === 20) + } + + testVector( + "SPARK-55897: ColumnarArray.get with struct-backed UDT", + 10, + new ArrayType(new StructType().add("x", IntegerType).add("y", LongType), false)) { column => + val data = column.arrayData() + data.getChild(0).putInt(0, 100) + data.getChild(1).putLong(0, 200L) + column.putArray(0, 0, 1) + + val arr = column.getArray(0) + val row = arr.get(0, new TestStructWrapperUDT()).asInstanceOf[InternalRow] + assert(row.getInt(0) === 100) + assert(row.getLong(1) === 200L) + } + + test("SPARK-55897: ColumnarBatchRow.get with primitive-backed UDT") { + Seq(MemoryMode.ON_HEAP, MemoryMode.OFF_HEAP).foreach { memMode => + val col = allocate(10, IntegerType, memMode) + try { + col.putInt(0, 99) + val batchRow = new ColumnarBatchRow(Array(col)) + batchRow.rowId = 0 + assert(batchRow.get(0, new TestIntUDT()) === 99) + } finally { + col.close() + } + } + } + + test("SPARK-55897: ColumnarBatchRow.get with struct-backed UDT") { + Seq(MemoryMode.ON_HEAP, MemoryMode.OFF_HEAP).foreach { memMode => + val col = allocate(10, + new StructType().add("x", IntegerType).add("y", LongType), memMode) + try { + col.getChild(0).putInt(0, 5) + col.getChild(1).putLong(0, 15L) + val batchRow = new ColumnarBatchRow(Array(col)) + batchRow.rowId = 0 + + val row = batchRow.get(0, new TestStructWrapperUDT()).asInstanceOf[InternalRow] + assert(row.getInt(0) === 5) + assert(row.getLong(1) === 15L) + } finally { + col.close() + } + } + } } From a985468ed74d4090d3af19300fd3dd83441cdf33 Mon Sep 17 00:00:00 2001 From: Ziya Mukhtarov Date: Mon, 11 May 2026 10:55:25 -0700 Subject: [PATCH 099/286] [SPARK-56551][SQL][FOLLOW-UP] Fix setting `numDeletedRows` metric as -1 ### What changes were proposed in this pull request? We were previously calling `SQLMetric.set(-1)` when we couldn't compute the value of `numDeletedRows` metric. However, this call was a no-op, and we reported this metric in the write summary as 0 instead. This PR fixes it to report -1 as intended. ### Why are the changes needed? Fix the bug above. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added a new test. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 Closes #55576 from ZiyaZa/fix-negative-numdeletedrows. Authored-by: Ziya Mukhtarov Signed-off-by: Gengliang Wang (cherry picked from commit 759036d6088b205f12f4f6073ce741896af42b10) Signed-off-by: Gengliang Wang --- .../v2/WriteToDataSourceV2Exec.scala | 91 ++++++++++--------- .../connector/DeleteFromTableSuiteBase.scala | 23 +++++ 2 files changed, 73 insertions(+), 41 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala index ccfcdc1855f04..3cbfed40d876e 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala @@ -365,24 +365,24 @@ case class ReplaceDataExec( copy(query = newChild) } - override protected def getWriteSummary(query: SparkPlan): Option[WriteSummary] = { - if (rowLevelCommand == DELETE) { - // DELETE ReplaceData plans filter out the deleted rows early in the plan, and they don't - // reach this node. We need to calculate this value as numScannedRows - numCopiedRows. - val numScannedRows = collectFirst(query) { - case b: BatchScanExec if b.table.isInstanceOf[RowLevelOperationTable] => - getMetricValue(b.metrics, "numOutputRows") - } - val numCopiedRows = getMetricValue(metrics, "numCopiedRows") - val numDeletedRows = if (numScannedRows.exists(_ >= 0) && numCopiedRows >= 0) { - numScannedRows.get - numCopiedRows - } else { - // One of the metrics couldn't be found, also mark numDeletedRows as not found. - -1L - } - metrics("numDeletedRows").set(numDeletedRows) + override protected def getDeleteSummary(): Option[DeleteSummaryImpl] = { + // DELETE ReplaceData plans filter out the deleted rows early in the plan, and they don't + // reach this node. We need to calculate this value as numScannedRows - numCopiedRows. + val numScannedRows = collectFirst(query) { + case b: BatchScanExec if b.table.isInstanceOf[RowLevelOperationTable] => + getMetricValue(b.metrics, "numOutputRows") } - super.getWriteSummary(query) + val numCopiedRows = getMetricValue(sparkMetrics, "numCopiedRows") + val numDeletedRows = if (numScannedRows.exists(_ >= 0) && numCopiedRows >= 0) { + numScannedRows.get - numCopiedRows + } else { + // One of the metrics couldn't be found, also mark numDeletedRows as not found. + -1L + } + + // SQLMetric.set is a no-op if value is -1, leaving the metric in its invalid state. + sparkMetrics("numDeletedRows").set(numDeletedRows) + super.getDeleteSummary().map(_.copy(numDeletedRows = numDeletedRows)) } } @@ -496,31 +496,40 @@ trait RowLevelWriteExec extends V2ExistingTableWriteExec { metrics.get(name).map(_.value).getOrElse(-1L) } - override protected def getWriteSummary(query: SparkPlan): Option[WriteSummary] = { + override protected def getWriteSummary(): Option[WriteSummary] = { rowLevelCommand match { - case MERGE => - collectFirst(query) { case m: MergeRowsExec => m }.map { n => - val metrics = n.metrics - MergeSummaryImpl( - getMetricValue(metrics, "numTargetRowsCopied"), - getMetricValue(metrics, "numTargetRowsDeleted"), - getMetricValue(metrics, "numTargetRowsUpdated"), - getMetricValue(metrics, "numTargetRowsInserted"), - getMetricValue(metrics, "numTargetRowsMatchedUpdated"), - getMetricValue(metrics, "numTargetRowsMatchedDeleted"), - getMetricValue(metrics, "numTargetRowsNotMatchedBySourceUpdated"), - getMetricValue(metrics, "numTargetRowsNotMatchedBySourceDeleted")) - } - case UPDATE => - Some(UpdateSummaryImpl( - getMetricValue(sparkMetrics, "numUpdatedRows"), - getMetricValue(sparkMetrics, "numCopiedRows"))) - case DELETE => - Some(DeleteSummaryImpl( - getMetricValue(sparkMetrics, "numDeletedRows"), - getMetricValue(sparkMetrics, "numCopiedRows"))) + case MERGE => getMergeSummary() + case UPDATE => getUpdateSummary() + case DELETE => getDeleteSummary() } } + + protected def getMergeSummary(): Option[MergeSummaryImpl] = { + collectFirst(query) { case m: MergeRowsExec => m }.map { n => + val metrics = n.metrics + MergeSummaryImpl( + getMetricValue(metrics, "numTargetRowsCopied"), + getMetricValue(metrics, "numTargetRowsDeleted"), + getMetricValue(metrics, "numTargetRowsUpdated"), + getMetricValue(metrics, "numTargetRowsInserted"), + getMetricValue(metrics, "numTargetRowsMatchedUpdated"), + getMetricValue(metrics, "numTargetRowsMatchedDeleted"), + getMetricValue(metrics, "numTargetRowsNotMatchedBySourceUpdated"), + getMetricValue(metrics, "numTargetRowsNotMatchedBySourceDeleted")) + } + } + + protected def getUpdateSummary(): Option[UpdateSummaryImpl] = { + Some(UpdateSummaryImpl( + getMetricValue(sparkMetrics, "numUpdatedRows"), + getMetricValue(sparkMetrics, "numCopiedRows"))) + } + + protected def getDeleteSummary(): Option[DeleteSummaryImpl] = { + Some(DeleteSummaryImpl( + getMetricValue(sparkMetrics, "numDeletedRows"), + getMetricValue(sparkMetrics, "numCopiedRows"))) + } } /** @@ -582,7 +591,7 @@ trait V2TableWriteExec } ) - val writeSummary = getWriteSummary(query) + val writeSummary = getWriteSummary() logInfo(log"Data source write support ${MDC(LogKeys.BATCH_WRITE, batchWrite)} is committing.") writeSummary match { case Some(summary) => batchWrite.commit(messages, summary) @@ -610,7 +619,7 @@ trait V2TableWriteExec Nil } - protected def getWriteSummary(query: SparkPlan): Option[WriteSummary] = None + protected def getWriteSummary(): Option[WriteSummary] = None } trait WritingSparkTask[W <: DataWriter[InternalRow]] extends Logging with Serializable { diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeleteFromTableSuiteBase.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeleteFromTableSuiteBase.scala index f8d81ee086911..89e3ce503fed0 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeleteFromTableSuiteBase.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeleteFromTableSuiteBase.scala @@ -959,6 +959,29 @@ abstract class DeleteFromTableSuiteBase extends RowLevelOperationSuiteBase { Row(2, 200, "software"))) } + test("delete with NOT IN over empty subquery") { + withTempView("empty_subq") { + createAndInitTable("pk INT NOT NULL, id INT NOT NULL, dep STRING", + """{ "pk": 1, "id": 1, "dep": "hr" } + |{ "pk": 2, "id": 2, "dep": "hr" } + |{ "pk": 3, "id": 3, "dep": "hr" } + |""".stripMargin) + + Seq.empty[Int].toDF("v").createOrReplaceTempView("empty_subq") + + sql( + s"""DELETE FROM $tableNameAsString + |WHERE id NOT IN (SELECT v FROM empty_subq) + |""".stripMargin) + + checkAnswer(sql(s"SELECT * FROM $tableNameAsString"), Nil) + // The filter gets replaced by an EmptyRelation in the ReplaceData executed plan, which hides + // the executed BatchScan and prevents computing numDeletedRows using numOutputRows of the + // scan node. + checkDeleteMetrics(numDeletedRows = if (deltaDelete) 3 else -1, numCopiedRows = 0) + } + } + private def executeDeleteWithFilters(query: String): Unit = { val executedPlan = executeAndKeepPlan { sql(query) From 23d271e91e4f421d7cccc6e3ca42af88fefd34b6 Mon Sep 17 00:00:00 2001 From: Gengliang Wang Date: Mon, 11 May 2026 11:36:50 -0700 Subject: [PATCH 100/286] [SPARK-56798][SQL][DOCS] Clarify streaming CDC emission timing and netChanges scope ### What changes were proposed in this pull request? Address two follow-up review threads on PR #55637 (streaming CDC netChanges) by clarifying the streaming behavior in the `Changelog` Javadoc. The previous paragraph read as if the emission lag were a netChanges-specific property; in fact carry-over removal and update detection use append-mode `Aggregate` keyed on `_commit_timestamp` and have the same lag as the netChanges `transformWithState` timer. The paragraph also did not set expectations for what streaming netChanges actually collapses in practice. Replaced the existing single paragraph with a bulleted list: - **Output is buffered until the watermark advances past the commit.** When a micro-batch ingests a commit, that commit's output rows are buffered in state and not emitted in the same batch. They are emitted by a later micro-batch -- whichever one advances the watermark past the commit's `_commit_timestamp`. The last commit's output is emitted when the source terminates. - **netChanges only merges changes that are buffered together.** When each row identity appears in at most one commit within any buffered window, the streaming output is the same as `computeUpdates`. Cross-commit merging only happens when several commits touch the same row before the earliest one's output has been released. For full-range collapse, use a batch read. This is a sub-task of SPARK-55668. ### Why are the changes needed? Spelling out the emission timing and the practical netChanges scope prevents adopters from forming wrong expectations about what streaming netChanges does for typical CDC workloads. Anchoring the lag on watermark progression (not commit count) and the netChanges merge condition on row-identity occurrences within the buffered window (not on changes within a single commit) keeps the doc consistent with what `CdcNetChangesStatefulProcessor` actually implements -- including the cases where multiple distinct commits share a micro-batch or the same row identity is touched in multiple commits before the older one's output has been released. ### Does this PR introduce _any_ user-facing change? Documentation only. No behavior change. ### How was this patch tested? Doc-only change. `Xdoclint:html,syntax,accessibility` is clean on `Changelog.java` (errors limited to expected "cannot find symbol" without classpath). No code changed; existing CDC test suites unaffected. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude opus-4-7 Closes #55776 from gengliangwang/SPARK-cdc-streaming-doc-clarify. Authored-by: Gengliang Wang Signed-off-by: Gengliang Wang (cherry picked from commit 5d47d6e4af0ac267b68bdcbb5df83ac5b393bb73) Signed-off-by: Gengliang Wang --- .../sql/connector/catalog/Changelog.java | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java index 2ef0846ab800b..2c1dc896c1ba6 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java @@ -71,10 +71,21 @@ * *

* Streaming reads support carry-over removal, update detection, and net change - * computation. Net change collapses are kept in the state store keyed by row identity; - * row identities only touched in the latest observed commit are held back until either a - * later commit (with strictly greater `_commit_timestamp`) advances the global watermark - * past them, or the source terminates. + * computation. Two streaming-specific behaviors to be aware of: + *

    + *
  • Output is buffered until the watermark advances past the commit. + * When a micro-batch ingests a commit, that commit's output rows are + * buffered in state and not emitted in the same batch. They are emitted + * by a later micro-batch -- whichever one advances the watermark past + * the commit's {@code _commit_timestamp}. The last commit's output is + * emitted when the source terminates.
    • + *
  • netChanges only merges changes that are buffered together. + * When each row identity appears in at most one commit within any + * buffered window, the streaming output is the same as + * {@code computeUpdates}. Cross-commit merging only happens when + * several commits touch the same row before the earliest one's output + * has been released. For full-range collapse, use a batch read.
    • + *
*

* Pushdown contract. When any post-processing pass applies (carry-over * removal, update detection, or netChanges), Spark only pushes predicates From 9f65c53b5e6673f2f62c10d9d09da8469104de40 Mon Sep 17 00:00:00 2001 From: Stanley Yao <592623+stanyao@users.noreply.github.com> Date: Tue, 12 May 2026 09:09:09 +0800 Subject: [PATCH 101/286] [SPARK-55978][SQL] Add TABLESAMPLE SYSTEM block sampling with DSv2 pushdown MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? This PR adds support for ANSI SQL `TABLESAMPLE SYSTEM` (block-level sampling) alongside the existing `TABLESAMPLE BERNOULLI` (row-level sampling). Key changes: - **SQL grammar**: Extended `TABLESAMPLE` to accept an optional `SYSTEM` or `BERNOULLI` qualifier before the sample method. Added both as non-reserved keywords. - **Logical plan**: Introduced `SampleMethod` sealed trait (`Bernoulli`/`System`) and added it to the `Sample` node. Default is `Bernoulli` for backward compatibility. - **Parser**: `TABLESAMPLE SYSTEM` only supports `PERCENT` sampling and does not support `REPEATABLE`. Other methods (ROWS, BUCKET, BYTES) are rejected with clear error messages. - **DSv2 pushdown**: `TABLESAMPLE SYSTEM` is pushed down to data sources via an extended `SupportsPushDownTableSample.pushTableSample()` overload with `isSystemSampling` flag. Sources that don't override the new method reject SYSTEM sampling by default. - **Physical planning**: SYSTEM samples that aren't pushed down to a DSv2 source raise an `AnalysisException` — there is no row-level fallback since block sampling is data-source dependent. ### Why are the changes needed? ANSI SQL defines two sampling methods: `BERNOULLI` (row-level) and `SYSTEM` (implementation-dependent, typically block/split-level). Block sampling is significantly faster for large tables since it avoids per-row evaluation, making it useful for approximate queries and data exploration. Many databases (PostgreSQL, Hive, Trino) support this distinction. ### Does this PR introduce _any_ user-facing change? Yes. New SQL syntax `TABLESAMPLE SYSTEM (x PERCENT)` and `TABLESAMPLE BERNOULLI (x PERCENT)`. `BERNOULLI` and `SYSTEM` are added as non-reserved keywords. Existing queries without these keywords behave identically to before. ### How was this patch tested? - 9 new test cases in `PlanParserSuite` covering: basic parsing, case insensitivity, boundary fractions, unsupported methods (ROWS/BUCKET with SYSTEM), REPEATABLE rejection, fraction validation, identifier preservation, and subquery contexts. - Existing `SQLQuerySuite` tests pass. - Scalastyle passes with 0 errors/warnings. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (thoroughly refined, reviewed, and tested by human) Closes #54972 from stanyao/spark-55978-tablesample-system. Lead-authored-by: Stanley Yao <592623+stanyao@users.noreply.github.com> Co-authored-by: Stanley Yao Signed-off-by: Wenchen Fan (cherry picked from commit 5fe5451c3680351ff96811477589617685d06cf6) Signed-off-by: Wenchen Fan --- .../resources/error/error-conditions.json | 20 ++ docs/sql-ref-ansi-compliance.md | 2 + .../spark/sql/catalyst/parser/SqlBaseLexer.g4 | 2 + .../sql/catalyst/parser/SqlBaseParser.g4 | 8 +- .../spark/sql/errors/QueryParsingErrors.scala | 16 ++ .../sql/connector/read/SampleMethod.java | 33 +++ .../read/SupportsPushDownTableSample.java | 19 +- .../UnsupportedOperationChecker.scala | 2 +- .../sql/catalyst/optimizer/Optimizer.scala | 2 +- .../sql/catalyst/parser/AstBuilder.scala | 25 +- .../plans/logical/basicLogicalOperators.scala | 22 +- .../sql/catalyst/parser/PlanParserSuite.scala | 201 ++++++++++++++ .../InMemoryTableWithTableSample.scala | 258 ++++++++++++++++++ .../InMemoryTableWithTableSampleCatalog.scala | 102 +++++++ .../SparkConnectDatabaseMetaDataSuite.scala | 4 +- .../sample_fraction_seed.explain | 2 +- ...mple_withReplacement_fraction_seed.explain | 2 +- .../sql/execution/DataSourceScanExec.scala | 8 +- .../spark/sql/execution/SparkStrategies.scala | 9 +- .../datasources/v2/PushDownUtils.scala | 9 +- .../datasources/v2/TableSampleInfo.scala | 5 +- .../v2/V2ScanRelationPushDown.scala | 23 +- .../analyzer-results/pipe-operators.sql.out | 12 +- .../results/keywords-enforced.sql.out | 2 + .../sql-tests/results/keywords.sql.out | 2 + .../results/nonansi/keywords.sql.out | 2 + .../DataSourceV2TableSampleSuite.scala | 210 ++++++++++++++ .../ThriftServerWithSparkContextSuite.scala | 2 +- 28 files changed, 977 insertions(+), 27 deletions(-) create mode 100644 sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SampleMethod.java create mode 100644 sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSample.scala create mode 100644 sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSampleCatalog.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2TableSampleSuite.scala diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json index 95f0c303e35fb..889ecf9f7b08a 100644 --- a/common/utils/src/main/resources/error/error-conditions.json +++ b/common/utils/src/main/resources/error/error-conditions.json @@ -8095,6 +8095,26 @@ "Store backend is not supported by TransformWithState operator. Please use RocksDBStateStoreProvider." ] }, + "TABLESAMPLE_SYSTEM" : { + "message" : [ + "TABLESAMPLE SYSTEM is only supported by data sources that implement block-level sampling." + ] + }, + "TABLESAMPLE_SYSTEM_NO_SCAN" : { + "message" : [ + "TABLESAMPLE SYSTEM requires a direct reference to a data source table that supports block-level sampling. It cannot be applied to subqueries, views, or tables with intervening operations." + ] + }, + "TABLESAMPLE_SYSTEM_REPEATABLE" : { + "message" : [ + "TABLESAMPLE SYSTEM does not support the REPEATABLE clause. Use TABLESAMPLE BERNOULLI for repeatable sampling with a seed." + ] + }, + "TABLESAMPLE_SYSTEM_SAMPLE_METHOD" : { + "message" : [ + "TABLESAMPLE SYSTEM does not support sampling. Only PERCENT sampling is supported." + ] + }, "TABLE_OPERATION" : { "message" : [ "Table does not support . Please check the current catalog and namespace to make sure the qualified table name is expected, and also check the catalog implementation which is configured by \"spark.sql.catalog\"." diff --git a/docs/sql-ref-ansi-compliance.md b/docs/sql-ref-ansi-compliance.md index 8542cd3d89865..4f21b7b4b3c79 100644 --- a/docs/sql-ref-ansi-compliance.md +++ b/docs/sql-ref-ansi-compliance.md @@ -430,6 +430,7 @@ Below is a list of all the keywords in Spark SQL. |ATOMIC|non-reserved|non-reserved|non-reserved| |AUTHORIZATION|reserved|non-reserved|reserved| |BEGIN|non-reserved|non-reserved|non-reserved| +|BERNOULLI|non-reserved|non-reserved|non-reserved| |BETWEEN|non-reserved|non-reserved|reserved| |BIGINT|non-reserved|non-reserved|reserved| |BINARY|non-reserved|non-reserved|reserved| @@ -765,6 +766,7 @@ Below is a list of all the keywords in Spark SQL. |SUBSTR|non-reserved|non-reserved|non-reserved| |SUBSTRING|non-reserved|non-reserved|non-reserved| |SYNC|non-reserved|non-reserved|non-reserved| +|SYSTEM|non-reserved|non-reserved|reserved| |SYSTEM_PATH|non-reserved|non-reserved|not a keyword| |SYSTEM_TIME|non-reserved|non-reserved|non-reserved| |SYSTEM_VERSION|non-reserved|non-reserved|non-reserved| diff --git a/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4 b/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4 index f4834b4ecf623..af71f441012c1 100644 --- a/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4 +++ b/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4 @@ -149,6 +149,7 @@ AT: 'AT'; ATOMIC: 'ATOMIC'; AUTHORIZATION: 'AUTHORIZATION'; BEGIN: 'BEGIN'; +BERNOULLI: 'BERNOULLI'; BETWEEN: 'BETWEEN'; BIGINT: 'BIGINT'; BINARY: 'BINARY'; @@ -483,6 +484,7 @@ STRUCT: 'STRUCT' {incComplexTypeLevelCounter();}; SUBSTR: 'SUBSTR'; SUBSTRING: 'SUBSTRING'; SYNC: 'SYNC'; +SYSTEM: 'SYSTEM'; SYSTEM_TIME: 'SYSTEM_TIME'; SYSTEM_VERSION: 'SYSTEM_VERSION'; SYSTEM_PATH: 'SYSTEM_PATH'; diff --git a/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 b/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 index 735921681cdcd..1e3acbc001b3c 100644 --- a/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 +++ b/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 @@ -1073,7 +1073,9 @@ nearestByClause ; sample - : TABLESAMPLE LEFT_PAREN sampleMethod? RIGHT_PAREN (REPEATABLE LEFT_PAREN seed=integerValue RIGHT_PAREN)? + : TABLESAMPLE (sampleType=(SYSTEM | BERNOULLI))? + LEFT_PAREN sampleMethod? RIGHT_PAREN + (REPEATABLE LEFT_PAREN seed=integerValue RIGHT_PAREN)? ; sampleMethod @@ -1942,6 +1944,7 @@ ansiNonReserved | AT | ATOMIC | BEGIN + | BERNOULLI | BETWEEN | BIGINT | BINARY @@ -2216,6 +2219,7 @@ ansiNonReserved | SUBSTR | SUBSTRING | SYNC + | SYSTEM | SYSTEM_PATH | SYSTEM_TIME | SYSTEM_VERSION @@ -2322,6 +2326,7 @@ nonReserved | ATOMIC | AUTHORIZATION | BEGIN + | BERNOULLI | BETWEEN | BIGINT | BINARY @@ -2645,6 +2650,7 @@ nonReserved | SUBSTR | SUBSTRING | SYNC + | SYSTEM | SYSTEM_PATH | SYSTEM_TIME | SYSTEM_VERSION diff --git a/sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala b/sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala index 33d7aaef17b81..eca7342a2d9e3 100644 --- a/sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala +++ b/sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala @@ -509,6 +509,22 @@ private[sql] object QueryParsingErrors extends DataTypeErrorsBase { ctx) } + def tableSampleSystemRepeatableError(ctx: ParserRuleContext): Throwable = { + new ParseException( + errorClass = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_REPEATABLE", + messageParameters = Map.empty, + ctx) + } + + def tableSampleSystemSampleMethodError( + sampleMethod: String, + ctx: ParserRuleContext): Throwable = { + new ParseException( + errorClass = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_SAMPLE_METHOD", + messageParameters = Map("sampleMethod" -> sampleMethod), + ctx) + } + def invalidStatementError(operation: String, ctx: ParserRuleContext): Throwable = { new ParseException( errorClass = "INVALID_STATEMENT_OR_CLAUSE", diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SampleMethod.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SampleMethod.java new file mode 100644 index 0000000000000..b9af8f9d5ac7f --- /dev/null +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SampleMethod.java @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector.read; + +import org.apache.spark.annotation.Evolving; + +/** + * The sampling method for TABLESAMPLE. + * + * @since 4.2.0 + */ +@Evolving +public enum SampleMethod { + /** Row-level sampling (BERNOULLI). Each row is independently selected. */ + BERNOULLI, + /** Block-level sampling (SYSTEM). Entire partitions/splits are included or skipped. */ + SYSTEM +} diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownTableSample.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownTableSample.java index 3630feb4680ea..3ceb7ed2de143 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownTableSample.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownTableSample.java @@ -29,11 +29,28 @@ public interface SupportsPushDownTableSample extends ScanBuilder { /** - * Pushes down SAMPLE to the data source. + * Pushes down BERNOULLI (row-level) SAMPLE to the data source. */ boolean pushTableSample( double lowerBound, double upperBound, boolean withReplacement, long seed); + + /** + * Pushes down SAMPLE to the data source with the specified sampling method. + */ + default boolean pushTableSample( + double lowerBound, + double upperBound, + boolean withReplacement, + long seed, + SampleMethod sampleMethod) { + if (sampleMethod == SampleMethod.SYSTEM) { + // If the data source hasn't overridden this method, it must not have added support + // for SYSTEM sampling. Don't apply sample pushdown. + return false; + } + return pushTableSample(lowerBound, upperBound, withReplacement, seed); + } } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala index b925d9da33423..83ad97fdc4faf 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala @@ -579,7 +579,7 @@ object UnsupportedOperationChecker extends Logging { throwError("Sorting is not supported on streaming DataFrames/Datasets, unless it is on " + "aggregated DataFrame/Dataset in Complete output mode") - case Sample(_, _, _, _, child) if child.isStreaming => + case Sample(_, _, _, _, child, _) if child.isStreaming => throwError("Sampling is not supported on streaming DataFrames/Datasets") case Window(windowExpression, _, _, child, _) if child.isStreaming => diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala index e4d53b697af80..ddfe80443d561 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala @@ -1296,7 +1296,7 @@ object CollapseProject extends Rule[LogicalPlan] with AliasHelper { limit.copy(child = p2.copy(projectList = newProjectList)) case Project(l1, r @ Repartition(_, _, p @ Project(l2, _))) if isRenaming(l1, l2) => r.copy(child = p.copy(projectList = buildCleanedProjectList(l1, p.projectList))) - case Project(l1, s @ Sample(_, _, _, _, p2 @ Project(l2, _))) if isRenaming(l1, l2) => + case Project(l1, s @ Sample(_, _, _, _, p2 @ Project(l2, _), _)) if isRenaming(l1, l2) => s.copy(child = p2.copy(projectList = buildCleanedProjectList(l1, p2.projectList))) case o => o } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala index 929fb2b4ceb15..95b21eb01b4b7 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala @@ -2448,10 +2448,14 @@ class AstBuilder extends DataTypeAstBuilder * - TABLESAMPLE(x ROWS): Sample the table down to the given number of rows. * - TABLESAMPLE(x PERCENT) [REPEATABLE (y)]: Sample the table down to the given percentage with * seed 'y'. Note that percentages are defined as a number between 0 and 100. + * - TABLESAMPLE SYSTEM(x PERCENT): Sample by data-source-dependent blocks or file splits. * - TABLESAMPLE(BUCKET x OUT OF y) [REPEATABLE (z)]: Sample the table down to a 'x' divided by * 'y' fraction with seed 'z'. */ private def withSample(ctx: SampleContext, query: LogicalPlan): LogicalPlan = withOrigin(ctx) { + val isSystem = ctx.sampleType != null && + ctx.sampleType.getType == SqlBaseParser.SYSTEM + // Create a sampled plan if we need one. def sample(fraction: Double, seed: Option[Long]): Sample = { // The range of fraction accepted by Sample is [0, 1]. Because Hive's block sampling @@ -2461,17 +2465,25 @@ class AstBuilder extends DataTypeAstBuilder validate(fraction >= 0.0 - eps && fraction <= 1.0 + eps, s"Sampling fraction ($fraction) must be on interval [0, 1]", ctx) - Sample(0.0, fraction, withReplacement = false, seed, query) + val method = if (isSystem) SampleMethod.System else SampleMethod.Bernoulli + Sample(0.0, fraction, withReplacement = false, seed, query, method) } if (ctx.sampleMethod() == null) { throw QueryParsingErrors.emptyInputForTableSampleError(ctx) } + if (isSystem && ctx.seed != null) { + throw QueryParsingErrors.tableSampleSystemRepeatableError(ctx) + } + val seed: Option[Long] = Option(ctx.seed).map(_.getText.toLong) ctx.sampleMethod() match { case ctx: SampleByRowsContext => + if (isSystem) { + throw QueryParsingErrors.tableSampleSystemSampleMethodError("ROWS", ctx) + } Limit(expression(ctx.expression), query) case ctx: SampleByPercentileContext => @@ -2483,6 +2495,9 @@ class AstBuilder extends DataTypeAstBuilder sample(sign * fraction / 100.0d, seed) case ctx: SampleByBytesContext => + if (isSystem) { + throw QueryParsingErrors.tableSampleSystemSampleMethodError("BYTES", ctx) + } val bytesStr = ctx.bytes.getText if (bytesStr.matches("[0-9]+[bBkKmMgG]")) { throw QueryParsingErrors.tableSampleByBytesUnsupportedError("byteLengthLiteral", ctx) @@ -2491,6 +2506,9 @@ class AstBuilder extends DataTypeAstBuilder } case ctx: SampleByBucketContext if ctx.ON() != null => + if (isSystem) { + throw QueryParsingErrors.tableSampleSystemSampleMethodError("BUCKET", ctx) + } if (ctx.identifier != null) { throw QueryParsingErrors.tableSampleByBytesUnsupportedError( "BUCKET x OUT OF y ON colname", ctx) @@ -2500,6 +2518,9 @@ class AstBuilder extends DataTypeAstBuilder } case ctx: SampleByBucketContext => + if (isSystem) { + throw QueryParsingErrors.tableSampleSystemSampleMethodError("BUCKET", ctx) + } sample(ctx.numerator.getText.toDouble / ctx.denominator.getText.toDouble, seed) } } @@ -2912,7 +2933,7 @@ class AstBuilder extends DataTypeAstBuilder // inline table comes in two styles: // style 1: values (1), (2), (3) -- multiple columns are supported // style 2: values 1, 2, 3 -- only a single column is supported here - // Strip Alias wrappers from row values — CreateStruct.apply preserves them for + // Strip Alias wrappers from row values - CreateStruct.apply preserves them for // expressions like `(1 AS id, 'a' AS name)`, but they are redundant here since // column names are determined by the table alias or generated defaults. case struct: CreateNamedStruct => struct.valExprs.map { diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala index 8e9f264698caf..6d37aa0f9f6b1 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala @@ -1912,6 +1912,14 @@ object SubqueryAlias { } } +sealed trait SampleMethod extends Serializable +object SampleMethod { + /** Row-level sampling (BERNOULLI). Each row independently selected. No I/O savings. */ + case object Bernoulli extends SampleMethod + /** System-level sampling (SYSTEM). Entire partitions/splits included or skipped. */ + case object System extends SampleMethod +} + object Sample { /** * Convenience constructor that wraps a concrete seed in [[Some]]. @@ -1926,6 +1934,16 @@ object Sample { child: LogicalPlan): Sample = { new Sample(lowerBound, upperBound, withReplacement, Some(seed), child) } + + def apply( + lowerBound: Double, + upperBound: Double, + withReplacement: Boolean, + seed: Long, + child: LogicalPlan, + sampleMethod: SampleMethod): Sample = { + new Sample(lowerBound, upperBound, withReplacement, Some(seed), child, sampleMethod) + } } /** @@ -1939,13 +1957,15 @@ object Sample { * (SQL `REPEATABLE` clause or programmatic API), `None` when no seed was * specified and a random seed should be generated at execution time. * @param child the LogicalPlan + * @param sampleMethod the sampling method (Bernoulli or System) */ case class Sample( lowerBound: Double, upperBound: Double, withReplacement: Boolean, seed: Option[Long], - child: LogicalPlan) extends UnaryNode { + child: LogicalPlan, + sampleMethod: SampleMethod = SampleMethod.Bernoulli) extends UnaryNode { val eps = RandomSampler.roundingEpsilon val fraction = upperBound - lowerBound diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala index 6124c69fbedda..1ecb7fa539c2c 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala @@ -1024,6 +1024,207 @@ class PlanParserSuite extends AnalysisTest { stop = 65)) } + test("SPARK-55978: TABLESAMPLE SYSTEM and BERNOULLI - basic parsing") { + val sql = "select * from t" + // SYSTEM produces SampleMethod.System + assertEqual( + s"$sql tablesample system (43 percent) as x", + Sample(0, .43d, withReplacement = false, None, + table("t").as("x"), SampleMethod.System).select(star())) + // BERNOULLI produces SampleMethod.Bernoulli + assertEqual( + s"$sql tablesample bernoulli (43 percent) as x", + Sample(0, .43d, withReplacement = false, None, + table("t").as("x"), SampleMethod.Bernoulli).select(star())) + // No qualifier defaults to Bernoulli (backward compat) + assertEqual( + s"$sql tablesample(43 percent) as x", + Sample(0, .43d, withReplacement = false, None, + table("t").as("x")).select(star())) + } + + test("SPARK-55978: TABLESAMPLE SYSTEM - case insensitivity") { + val sql = "select * from t" + // Keywords are case-insensitive + assertEqual( + s"$sql TABLESAMPLE SYSTEM (43 PERCENT) as x", + Sample(0, .43d, withReplacement = false, None, + table("t").as("x"), SampleMethod.System).select(star())) + assertEqual( + s"$sql TabLeSaMpLe SyStEm (43 PeRcEnT) as x", + Sample(0, .43d, withReplacement = false, None, + table("t").as("x"), SampleMethod.System).select(star())) + assertEqual( + s"$sql TABLESAMPLE BERNOULLI (43 PERCENT) as x", + Sample(0, .43d, withReplacement = false, None, + table("t").as("x"), SampleMethod.Bernoulli).select(star())) + } + + test("SPARK-55978: TABLESAMPLE SYSTEM - boundary fractions") { + val sql = "select * from t" + // 0 PERCENT + assertEqual( + s"$sql tablesample system (0 percent) as x", + Sample(0, 0d, withReplacement = false, None, + table("t").as("x"), SampleMethod.System).select(star())) + // 100 PERCENT + assertEqual( + s"$sql tablesample system (100 percent) as x", + Sample(0, 1d, withReplacement = false, None, + table("t").as("x"), SampleMethod.System).select(star())) + // Fractional percent + assertEqual( + s"$sql tablesample system (0.1 percent) as x", + Sample(0, 0.001d, withReplacement = false, None, + table("t").as("x"), SampleMethod.System).select(star())) + } + + test("SPARK-55978: TABLESAMPLE SYSTEM - unsupported sample methods") { + val sql = "select * from t" + // SYSTEM + ROWS -> error + checkError( + exception = parseException(s"$sql tablesample system (100 rows)"), + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_SAMPLE_METHOD", + sqlState = "0A000", + parameters = Map("sampleMethod" -> "ROWS"), + context = ExpectedContext( + fragment = "tablesample system (100 rows)", + start = 16, + stop = 44)) + // SYSTEM + BYTES -> error + checkError( + exception = parseException(s"$sql tablesample system (300M)"), + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_SAMPLE_METHOD", + sqlState = "0A000", + parameters = Map("sampleMethod" -> "BYTES"), + context = ExpectedContext( + fragment = "tablesample system (300M)", + start = 16, + stop = 40)) + // SYSTEM + BUCKET -> error + checkError( + exception = parseException(s"$sql tablesample system (bucket 4 out of 10)"), + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_SAMPLE_METHOD", + sqlState = "0A000", + parameters = Map("sampleMethod" -> "BUCKET"), + context = ExpectedContext( + fragment = "tablesample system (bucket 4 out of 10)", + start = 16, + stop = 54)) + // SYSTEM + BUCKET ON colname -> error + checkError( + exception = parseException(s"$sql tablesample system (bucket 4 out of 10 on x)"), + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_SAMPLE_METHOD", + sqlState = "0A000", + parameters = Map("sampleMethod" -> "BUCKET"), + context = ExpectedContext( + fragment = "tablesample system (bucket 4 out of 10 on x)", + start = 16, + stop = 59)) + // SYSTEM + BUCKET ON function -> error + checkError( + exception = parseException(s"$sql tablesample system (bucket 3 out of 32 on rand())"), + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_SAMPLE_METHOD", + sqlState = "0A000", + parameters = Map("sampleMethod" -> "BUCKET"), + context = ExpectedContext( + fragment = "tablesample system (bucket 3 out of 32 on rand())", + start = 16, + stop = 64)) + } + + test("SPARK-55978: TABLESAMPLE BERNOULLI - REPEATABLE is supported") { + assertEqual( + "select * from t tablesample bernoulli (43 percent) repeatable (123) as x", + Sample(0, .43d, withReplacement = false, 123L, + table("t").as("x"), SampleMethod.Bernoulli).select(star())) + } + + test("SPARK-55978: TABLESAMPLE SYSTEM - REPEATABLE not supported") { + val sql = "select * from t" + checkError( + exception = parseException(s"$sql tablesample system (43 percent) repeatable (123)"), + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_REPEATABLE", + sqlState = "0A000", + context = ExpectedContext( + fragment = "tablesample system (43 percent) repeatable (123)", + start = 16, + stop = 63)) + } + + test("SPARK-55978: TABLESAMPLE SYSTEM - fraction out of range") { + val sql = "select * from t" + // > 100 PERCENT + checkError( + exception = parseException(s"$sql tablesample system (150 percent) as x"), + condition = "_LEGACY_ERROR_TEMP_0064", + parameters = Map("msg" -> "Sampling fraction (1.5) must be on interval [0, 1]"), + context = ExpectedContext( + fragment = "tablesample system (150 percent)", + start = 16, + stop = 47)) + // Negative PERCENT + checkError( + exception = parseException(s"$sql tablesample system (-10 percent) as x"), + condition = "_LEGACY_ERROR_TEMP_0064", + parameters = Map("msg" -> "Sampling fraction (-0.1) must be on interval [0, 1]"), + context = ExpectedContext( + fragment = "tablesample system (-10 percent)", + start = 16, + stop = 47)) + } + + test("SPARK-55978: TABLESAMPLE SYSTEM and BERNOULLI as identifiers") { + // SYSTEM usable as column name (nonReserved) + assertEqual("SELECT system FROM t", + table("t").select($"system")) + // BERNOULLI usable as column name + assertEqual("SELECT bernoulli FROM t", + table("t").select($"bernoulli")) + // Usable as table alias + assertEqual("SELECT * FROM t system", + table("t").as("system").select(star())) + assertEqual("SELECT * FROM t bernoulli", + table("t").as("bernoulli").select(star())) + // SYSTEM as table name with default (Bernoulli) TABLESAMPLE + assertEqual("SELECT * FROM system TABLESAMPLE(10 PERCENT) AS x", + Sample(0, .1d, withReplacement = false, None, + table("system").as("x")).select(star())) + // SYSTEM as table name with TABLESAMPLE SYSTEM qualifier + assertEqual("SELECT * FROM system TABLESAMPLE SYSTEM (10 PERCENT) AS x", + Sample(0, .1d, withReplacement = false, None, + table("system").as("x"), SampleMethod.System).select(star())) + // SYSTEM as both table name and alias with TABLESAMPLE + assertEqual("SELECT * FROM system TABLESAMPLE(10 PERCENT) system", + Sample(0, .1d, withReplacement = false, None, + table("system").as("system")).select(star())) + // BERNOULLI as table name with TABLESAMPLE BERNOULLI qualifier + assertEqual("SELECT * FROM bernoulli TABLESAMPLE BERNOULLI (10 PERCENT) AS x", + Sample(0, .1d, withReplacement = false, None, + table("bernoulli").as("x"), SampleMethod.Bernoulli).select(star())) + // SYSTEM as table name with TABLESAMPLE BERNOULLI (cross-keyword) + assertEqual("SELECT * FROM system TABLESAMPLE BERNOULLI (10 PERCENT) AS x", + Sample(0, .1d, withReplacement = false, None, + table("system").as("x"), SampleMethod.Bernoulli).select(star())) + // BERNOULLI as both table name and alias with TABLESAMPLE + assertEqual("SELECT * FROM bernoulli TABLESAMPLE(10 PERCENT) bernoulli", + Sample(0, .1d, withReplacement = false, None, + table("bernoulli").as("bernoulli")).select(star())) + // Schema-qualified SYSTEM table name with TABLESAMPLE SYSTEM + assertEqual("SELECT * FROM mydb.system TABLESAMPLE SYSTEM (10 PERCENT) AS x", + Sample(0, .1d, withReplacement = false, None, + table("mydb", "system").as("x"), SampleMethod.System).select(star())) + } + + test("SPARK-55978: TABLESAMPLE SYSTEM - subquery and join contexts") { + // SYSTEM sample in subquery + assertEqual( + "SELECT * FROM (SELECT * FROM t TABLESAMPLE SYSTEM (50 PERCENT)) sub", + Sample(0, .5d, withReplacement = false, None, + table("t"), SampleMethod.System) + .select(star()).as("sub").select(star())) + } + test("sub-query") { val plan = table("t0").select($"id") assertEqual("select id from (t0)", plan) diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSample.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSample.scala new file mode 100644 index 0000000000000..514a7f3beda40 --- /dev/null +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSample.scala @@ -0,0 +1,258 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector.catalog + +import java.util +import java.util.Locale + +import org.apache.spark.sql.connector.expressions.Transform +import org.apache.spark.sql.connector.expressions.filter.Predicate +import org.apache.spark.sql.connector.join.JoinType +import org.apache.spark.sql.connector.read.{InputPartition, SampleMethod, Scan, ScanBuilder, SupportsPushDownJoin, SupportsPushDownTableSample, SupportsPushDownV2Filters} +import org.apache.spark.sql.connector.read.SupportsPushDownJoin.ColumnWithAlias +import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder} +import org.apache.spark.sql.sources.Filter +import org.apache.spark.sql.types.StructType +import org.apache.spark.sql.util.CaseInsensitiveStringMap +import org.apache.spark.util.ArrayImplicits._ + +/** + * An in-memory table that supports TABLESAMPLE pushdown (both BERNOULLI and SYSTEM). + * + * For SYSTEM sampling, entire splits (InputPartitions) are included or skipped based on + * a hash of their index and the seed. For BERNOULLI sampling, the pushdown is accepted + * but rows are not actually filtered (Spark's row-level Sample operator handles it). + */ +class InMemoryTableWithTableSample( + name: String, + columns: Array[Column], + partitioning: Array[Transform], + properties: util.Map[String, String]) + extends InMemoryBaseTable(name, columns, partitioning, properties) { + + override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = { + InMemoryBaseTable.maybeSimulateFailedTableWrite(new CaseInsensitiveStringMap(properties)) + InMemoryBaseTable.maybeSimulateFailedTableWrite(info.options) + new InMemoryWriterBuilder(info) { + override def truncate(): WriteBuilder = { + writer = new TruncateAndAppend(this.info) + streamingWriter = new StreamingTruncateAndAppend(this.info) + this + } + } + } + + override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = { + new InMemoryTableSampleScanBuilder(schema, options) + } + + class InMemoryTableSampleScanBuilder( + tableSchema: StructType, + options: CaseInsensitiveStringMap) + extends InMemoryScanBuilder(tableSchema, options) with SupportsPushDownTableSample { + + private var sampleFraction: Double = 1.0 + private var sampleSeed: Long = 0L + private var sampleMethod: SampleMethod = SampleMethod.BERNOULLI + private var sampleWithReplacement: Boolean = false + private var samplePushed: Boolean = false + + override def pushTableSample( + lowerBound: Double, + upperBound: Double, + withReplacement: Boolean, + seed: Long): Boolean = { + this.sampleFraction = upperBound - lowerBound + this.sampleSeed = seed + this.sampleMethod = SampleMethod.BERNOULLI + this.sampleWithReplacement = withReplacement + this.samplePushed = true + true + } + + override def pushTableSample( + lowerBound: Double, + upperBound: Double, + withReplacement: Boolean, + seed: Long, + sampleMethod: SampleMethod): Boolean = { + this.sampleFraction = upperBound - lowerBound + this.sampleSeed = seed + this.sampleMethod = sampleMethod + this.sampleWithReplacement = withReplacement + this.samplePushed = true + true + } + + override def build: Scan = { + val allPartitions = data.map(_.asInstanceOf[InputPartition]).toImmutableArraySeq + val filteredPartitions = if (samplePushed && sampleMethod == SampleMethod.SYSTEM) { + // SYSTEM sampling: include/skip entire splits based on hash of index + seed + allPartitions.zipWithIndex.filter { case (_, idx) => + val hash = ((idx.toLong * 31 + sampleSeed) & Long.MaxValue).toDouble / Long.MaxValue + hash < sampleFraction + }.map(_._1) + } else { + allPartitions + } + if (samplePushed) { + new InMemoryBatchScanWithSample( + filteredPartitions, schema, tableSchema, options, + sampleFraction, sampleSeed, sampleMethod, sampleWithReplacement) + } else { + InMemoryBatchScan(filteredPartitions, schema, tableSchema, options) + } + } + } + + private class InMemoryBatchScanWithSample( + data: Seq[InputPartition], + readSchema: StructType, + tableSchema: StructType, + options: CaseInsensitiveStringMap, + sampleFraction: Double, + sampleSeed: Long, + sampleMethod: SampleMethod, + sampleWithReplacement: Boolean) + extends InMemoryBatchScan(data, readSchema, tableSchema, options) { + + override def description(): String = { + val pct = sampleFraction * 100 + val method = sampleMethod.toString.toUpperCase(Locale.ROOT) + s"${super.description()} $method SAMPLE ($pct) $sampleWithReplacement SEED($sampleSeed)" + } + } +} + +/** + * An in-memory table that supports both TABLESAMPLE pushdown and JOIN pushdown. + * Used to test the guard that prevents join pushdown when a side has a pushed sample. + */ +class InMemoryTableWithJoinAndSample( + name: String, + columns: Array[Column], + partitioning: Array[Transform], + properties: util.Map[String, String]) + extends InMemoryTableWithTableSample(name, columns, partitioning, properties) { + + override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = { + new InMemoryJoinAndSampleScanBuilder(schema, options) + } + + class InMemoryJoinAndSampleScanBuilder( + tableSchema: StructType, + options: CaseInsensitiveStringMap) + extends InMemoryTableSampleScanBuilder(tableSchema, options) + with SupportsPushDownJoin with SupportsPushDownV2Filters { + + private[catalog] val ownSchema: StructType = tableSchema + private var pushed: Array[Predicate] = Array.empty + private var joinedSchema: Option[StructType] = None + + override def pushPredicates(predicates: Array[Predicate]): Array[Predicate] = { + pushed = predicates + // Return empty - all predicates accepted (not actually filtered, just cleared + // so that the join pushdown pattern's Nil filter requirement is satisfied). + Array.empty + } + + // Override V1 pushFilters (inherited from InMemoryScanBuilder) to also accept all + // filters. PushDownUtils.pushFilters matches SupportsPushDownFilters before + // SupportsPushDownV2Filters, so without this override isnotnull predicates remain + // as post-scan Filter nodes and block the join pushdown pattern match. + override def pushFilters(filters: Array[Filter]): Array[Filter] = { + Array.empty + } + + override def pushedPredicates(): Array[Predicate] = pushed + + override def isOtherSideCompatibleForJoin(other: SupportsPushDownJoin): Boolean = true + + override def pushDownJoin( + other: SupportsPushDownJoin, + joinType: JoinType, + leftSideRequiredColumnsWithAliases: Array[ColumnWithAlias], + rightSideRequiredColumnsWithAliases: Array[ColumnWithAlias], + condition: Predicate): Boolean = { + val otherSchema = other.asInstanceOf[InMemoryJoinAndSampleScanBuilder].ownSchema + val leftFields = leftSideRequiredColumnsWithAliases.map { col => + val name = if (col.alias() != null) col.alias() else col.colName() + tableSchema(col.colName()).copy(name = name) + } + val rightFields = rightSideRequiredColumnsWithAliases.map { col => + val name = if (col.alias() != null) col.alias() else col.colName() + otherSchema(col.colName()).copy(name = name) + } + joinedSchema = Some(StructType(leftFields ++ rightFields)) + true + } + + override def build: Scan = { + joinedSchema match { + case Some(js) => + InMemoryBatchScan( + data.map(_.asInstanceOf[InputPartition]).toImmutableArraySeq, + js, tableSchema, options) + case None => super.build + } + } + } +} + +/** + * An in-memory table that supports TABLESAMPLE pushdown using only the legacy 4-arg + * pushTableSample method (does NOT override the 5-arg default). Used to test backward + * compatibility: BERNOULLI should push down via the default delegation, and SYSTEM + * should fail because the default returns false for SYSTEM. + */ +class InMemoryTableWithLegacyTableSample( + name: String, + columns: Array[Column], + partitioning: Array[Transform], + properties: util.Map[String, String]) + extends InMemoryBaseTable(name, columns, partitioning, properties) { + + override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = { + InMemoryBaseTable.maybeSimulateFailedTableWrite(new CaseInsensitiveStringMap(properties)) + InMemoryBaseTable.maybeSimulateFailedTableWrite(info.options) + new InMemoryWriterBuilder(info) { + override def truncate(): WriteBuilder = { + writer = new TruncateAndAppend(this.info) + streamingWriter = new StreamingTruncateAndAppend(this.info) + this + } + } + } + + override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = { + new InMemoryLegacySampleScanBuilder(schema, options) + } + + class InMemoryLegacySampleScanBuilder( + tableSchema: StructType, + options: CaseInsensitiveStringMap) + extends InMemoryScanBuilder(tableSchema, options) with SupportsPushDownTableSample { + + // Only the 4-arg method is overridden; the 5-arg default method is inherited. + override def pushTableSample( + lowerBound: Double, + upperBound: Double, + withReplacement: Boolean, + seed: Long): Boolean = true + } +} diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSampleCatalog.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSampleCatalog.scala new file mode 100644 index 0000000000000..12da978ea11a0 --- /dev/null +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSampleCatalog.scala @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector.catalog + +import java.util + +import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException +import org.apache.spark.sql.connector.expressions.Transform + +class InMemoryTableWithTableSampleCatalog extends InMemoryTableCatalog { + import CatalogV2Implicits._ + + override def createTable( + ident: Identifier, + columns: Array[Column], + partitions: Array[Transform], + properties: util.Map[String, String]): Table = { + if (tables.containsKey(ident)) { + throw new TableAlreadyExistsException(ident.asMultipartIdentifier) + } + + InMemoryTableCatalog.maybeSimulateFailedTableCreation(properties) + + val tableName = s"$name.${ident.quoted}" + val table = new InMemoryTableWithTableSample(tableName, columns, partitions, properties) + tables.put(ident, table) + namespaces.putIfAbsent(ident.namespace.toList, Map()) + table + } + + override def createTable(ident: Identifier, tableInfo: TableInfo): Table = { + createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties) + } +} + +class InMemoryTableWithJoinAndSampleCatalog extends InMemoryTableCatalog { + import CatalogV2Implicits._ + + override def createTable( + ident: Identifier, + columns: Array[Column], + partitions: Array[Transform], + properties: util.Map[String, String]): Table = { + if (tables.containsKey(ident)) { + throw new TableAlreadyExistsException(ident.asMultipartIdentifier) + } + + InMemoryTableCatalog.maybeSimulateFailedTableCreation(properties) + + val tableName = s"$name.${ident.quoted}" + val table = new InMemoryTableWithJoinAndSample(tableName, columns, partitions, properties) + tables.put(ident, table) + namespaces.putIfAbsent(ident.namespace.toList, Map()) + table + } + + override def createTable(ident: Identifier, tableInfo: TableInfo): Table = { + createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties) + } +} + +class InMemoryTableWithLegacyTableSampleCatalog extends InMemoryTableCatalog { + import CatalogV2Implicits._ + + override def createTable( + ident: Identifier, + columns: Array[Column], + partitions: Array[Transform], + properties: util.Map[String, String]): Table = { + if (tables.containsKey(ident)) { + throw new TableAlreadyExistsException(ident.asMultipartIdentifier) + } + + InMemoryTableCatalog.maybeSimulateFailedTableCreation(properties) + + val tableName = s"$name.${ident.quoted}" + val table = new InMemoryTableWithLegacyTableSample( + tableName, columns, partitions, properties) + tables.put(ident, table) + namespaces.putIfAbsent(ident.namespace.toList, Map()) + table + } + + override def createTable(ident: Identifier, tableInfo: TableInfo): Table = { + createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties) + } +} diff --git a/sql/connect/client/jdbc/src/test/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaDataSuite.scala b/sql/connect/client/jdbc/src/test/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaDataSuite.scala index 1cfe05a2b5c1f..1f525a541daae 100644 --- a/sql/connect/client/jdbc/src/test/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaDataSuite.scala +++ b/sql/connect/client/jdbc/src/test/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaDataSuite.scala @@ -209,8 +209,8 @@ class SparkConnectDatabaseMetaDataSuite extends ConnectFunSuite with RemoteSpark withConnection { conn => val metadata = conn.getMetaData // scalastyle:off line.size.limit - // CURRENT_PATH is excluded: getSQLKeywords drops SQL:2003 reserved words (see companion). - assert(metadata.getSQLKeywords === "ADD,AFTER,AGGREGATE,ALWAYS,ANALYZE,ANTI,ANY_VALUE,APPROX,ARCHIVE,ASC,BINDING,BUCKET,BUCKETS,BYTE,CACHE,CASCADE,CATALOG,CATALOGS,CHANGE,CHANGES,CLEAR,CLUSTER,CLUSTERED,CODEGEN,COLLATION,COLLATIONS,COLLECTION,COLUMNS,COMMENT,COMPACT,COMPACTIONS,COMPENSATION,COMPUTE,CONCATENATE,CONTAINS,CONTINUE,COST,CURRENT_DATABASE,CURRENT_SCHEMA,DATA,DATABASE,DATABASES,DATEADD,DATEDIFF,DATE_ADD,DATE_DIFF,DAYOFYEAR,DAYS,DBPROPERTIES,DEFAULT_PATH,DEFINED,DEFINER,DELAY,DELIMITED,DESC,DFS,DIRECTORIES,DIRECTORY,DISTANCE,DISTRIBUTE,DIV,DO,ELSEIF,ENFORCED,ESCAPED,EVOLUTION,EXACT,EXCHANGE,EXCLUDE,EXCLUSIVE,EXIT,EXPLAIN,EXPORT,EXTEND,EXTENDED,FIELDS,FILEFORMAT,FIRST,FLOW,FOLLOWING,FORMAT,FORMATTED,FOUND,FUNCTIONS,GENERATED,GEOGRAPHY,GEOMETRY,HANDLER,HOURS,IDENTIFIED,IDENTIFIER,IF,IGNORE,ILIKE,IMMEDIATE,INCLUDE,INCLUSIVE,INCREMENT,INDEX,INDEXES,INPATH,INPUT,INPUTFORMAT,INVOKER,ITEMS,ITERATE,JSON,KEY,KEYS,LAST,LAZY,LEAVE,LEVEL,LIMIT,LINES,LIST,LOAD,LOCATION,LOCK,LOCKS,LOGICAL,LONG,LOOP,MACRO,MAP,MATCHED,MATERIALIZED,MEASURE,METRICS,MICROSECOND,MICROSECONDS,MILLISECOND,MILLISECONDS,MINUS,MINUTES,MONTHS,MSCK,NAME,NAMESPACE,NAMESPACES,NANOSECOND,NANOSECONDS,NEAREST,NORELY,NULLS,OFFSET,OPTION,OPTIONS,OUTPUTFORMAT,OVERWRITE,PARTITIONED,PARTITIONS,PATH,PERCENT,PIVOT,PLACING,PRECEDING,PRINCIPALS,PROCEDURES,PROPERTIES,PURGE,QUALIFY,QUARTER,QUERY,RECORDREADER,RECORDWRITER,RECOVER,RECURSION,REDUCE,REFRESH,RELY,RENAME,REPAIR,REPEAT,REPEATABLE,REPLACE,RESET,RESPECT,RESTRICT,ROLE,ROLES,SCHEMA,SCHEMAS,SECONDS,SECURITY,SEMI,SEPARATED,SERDE,SERDEPROPERTIES,SETS,SHORT,SHOW,SIMILARITY,SINGLE,SKEWED,SORT,SORTED,SOURCE,STATISTICS,STORED,STRATIFY,STREAM,STREAMING,STRING,STRUCT,SUBSTR,SYNC,SYSTEM_PATH,SYSTEM_TIME,SYSTEM_VERSION,TABLES,TARGET,TBLPROPERTIES,TERMINATED,TIMEDIFF,TIMESTAMPADD,TIMESTAMPDIFF,TIMESTAMP_LTZ,TIMESTAMP_NTZ,TINYINT,TOUCH,TRANSACTION,TRANSACTIONS,TRANSFORM,TRUNCATE,TRY_CAST,TYPE,UNARCHIVE,UNBOUNDED,UNCACHE,UNLOCK,UNPIVOT,UNSET,UNTIL,USE,VAR,VARIABLE,VARIANT,VERSION,VIEW,VIEWS,VOID,WATERMARK,WEEK,WEEKS,WHILE,X,YEARS,ZONE") + // CURRENT_PATH and SYSTEM are excluded: getSQLKeywords drops SQL:2003 reserved words (see companion). + assert(metadata.getSQLKeywords === "ADD,AFTER,AGGREGATE,ALWAYS,ANALYZE,ANTI,ANY_VALUE,APPROX,ARCHIVE,ASC,BERNOULLI,BINDING,BUCKET,BUCKETS,BYTE,CACHE,CASCADE,CATALOG,CATALOGS,CHANGE,CHANGES,CLEAR,CLUSTER,CLUSTERED,CODEGEN,COLLATION,COLLATIONS,COLLECTION,COLUMNS,COMMENT,COMPACT,COMPACTIONS,COMPENSATION,COMPUTE,CONCATENATE,CONTAINS,CONTINUE,COST,CURRENT_DATABASE,CURRENT_SCHEMA,DATA,DATABASE,DATABASES,DATEADD,DATEDIFF,DATE_ADD,DATE_DIFF,DAYOFYEAR,DAYS,DBPROPERTIES,DEFAULT_PATH,DEFINED,DEFINER,DELAY,DELIMITED,DESC,DFS,DIRECTORIES,DIRECTORY,DISTANCE,DISTRIBUTE,DIV,DO,ELSEIF,ENFORCED,ESCAPED,EVOLUTION,EXACT,EXCHANGE,EXCLUDE,EXCLUSIVE,EXIT,EXPLAIN,EXPORT,EXTEND,EXTENDED,FIELDS,FILEFORMAT,FIRST,FLOW,FOLLOWING,FORMAT,FORMATTED,FOUND,FUNCTIONS,GENERATED,GEOGRAPHY,GEOMETRY,HANDLER,HOURS,IDENTIFIED,IDENTIFIER,IF,IGNORE,ILIKE,IMMEDIATE,INCLUDE,INCLUSIVE,INCREMENT,INDEX,INDEXES,INPATH,INPUT,INPUTFORMAT,INVOKER,ITEMS,ITERATE,JSON,KEY,KEYS,LAST,LAZY,LEAVE,LEVEL,LIMIT,LINES,LIST,LOAD,LOCATION,LOCK,LOCKS,LOGICAL,LONG,LOOP,MACRO,MAP,MATCHED,MATERIALIZED,MEASURE,METRICS,MICROSECOND,MICROSECONDS,MILLISECOND,MILLISECONDS,MINUS,MINUTES,MONTHS,MSCK,NAME,NAMESPACE,NAMESPACES,NANOSECOND,NANOSECONDS,NEAREST,NORELY,NULLS,OFFSET,OPTION,OPTIONS,OUTPUTFORMAT,OVERWRITE,PARTITIONED,PARTITIONS,PATH,PERCENT,PIVOT,PLACING,PRECEDING,PRINCIPALS,PROCEDURES,PROPERTIES,PURGE,QUALIFY,QUARTER,QUERY,RECORDREADER,RECORDWRITER,RECOVER,RECURSION,REDUCE,REFRESH,RELY,RENAME,REPAIR,REPEAT,REPEATABLE,REPLACE,RESET,RESPECT,RESTRICT,ROLE,ROLES,SCHEMA,SCHEMAS,SECONDS,SECURITY,SEMI,SEPARATED,SERDE,SERDEPROPERTIES,SETS,SHORT,SHOW,SIMILARITY,SINGLE,SKEWED,SORT,SORTED,SOURCE,STATISTICS,STORED,STRATIFY,STREAM,STREAMING,STRING,STRUCT,SUBSTR,SYNC,SYSTEM_PATH,SYSTEM_TIME,SYSTEM_VERSION,TABLES,TARGET,TBLPROPERTIES,TERMINATED,TIMEDIFF,TIMESTAMPADD,TIMESTAMPDIFF,TIMESTAMP_LTZ,TIMESTAMP_NTZ,TINYINT,TOUCH,TRANSACTION,TRANSACTIONS,TRANSFORM,TRUNCATE,TRY_CAST,TYPE,UNARCHIVE,UNBOUNDED,UNCACHE,UNLOCK,UNPIVOT,UNSET,UNTIL,USE,VAR,VARIABLE,VARIANT,VERSION,VIEW,VIEWS,VOID,WATERMARK,WEEK,WEEKS,WHILE,X,YEARS,ZONE") // scalastyle:on line.size.limit } } diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/sample_fraction_seed.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/sample_fraction_seed.explain index f94e0a850e403..9bcbf88135399 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/sample_fraction_seed.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/sample_fraction_seed.explain @@ -1,2 +1,2 @@ -Sample 0.0, 0.43, false, 9890823 +Sample 0.0, 0.43, false, 9890823, Bernoulli +- LocalRelation , [id#0L, a#0, b#0] diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/sample_withReplacement_fraction_seed.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/sample_withReplacement_fraction_seed.explain index 340c25ab6d017..5af5314e48f90 100644 --- a/sql/connect/common/src/test/resources/query-tests/explain-results/sample_withReplacement_fraction_seed.explain +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/sample_withReplacement_fraction_seed.explain @@ -1,2 +1,2 @@ -Sample 0.0, 0.23, true, 898 +Sample 0.0, 0.23, true, 898, Bernoulli +- LocalRelation , [id#0L, a#0, b#0] diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala index 2488b6aa51159..be7013188f2f9 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala @@ -17,6 +17,7 @@ package org.apache.spark.sql.execution +import java.util.Locale import java.util.concurrent.TimeUnit._ import org.apache.hadoop.fs.Path @@ -159,8 +160,11 @@ case class RowDataSourceScanExec( private def seqToString(seq: Seq[Any]): String = seq.mkString("[", ", ", "]") - private def pushedSampleMetadataString(s: TableSampleInfo): String = - s"SAMPLE (${(s.upperBound - s.lowerBound) * 100}) ${s.withReplacement} SEED(${s.seed})" + private def pushedSampleMetadataString(s: TableSampleInfo): String = { + val pct = (s.upperBound - s.lowerBound) * 100 + val method = s.sampleMethod.toString.toUpperCase(Locale.ROOT) + s"$method SAMPLE ($pct) ${s.withReplacement} SEED(${s.seed})" + } override val metadata: Map[String, String] = { val markedFilters = if (filters.nonEmpty) { diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala index 2c060aa3f9a5b..92818c12bfa09 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala @@ -1040,7 +1040,14 @@ abstract class SparkStrategies extends QueryPlanner[SparkPlan] { execution.FilterExec(f.typedCondition(f.deserializer), planLater(f.child)) :: Nil case e @ logical.Expand(_, _, child) => execution.ExpandExec(e.projections, e.output, planLater(child)) :: Nil - case logical.Sample(lb, ub, withReplacement, seed, child) => + case logical.Sample(lb, ub, withReplacement, seed, child, sampleMethod) => + if (sampleMethod == logical.SampleMethod.System) { + // V2ScanRelationPushDown is non-excludable and always handles SYSTEM samples + // (either pushes down or throws). Reaching here indicates an internal invariant + // violation. + throw SparkException.internalError( + "TABLESAMPLE SYSTEM node was not properly handled by V2ScanRelationPushDown.") + } execution.SampleExec(lb, ub, withReplacement, seed, planLater(child)) :: Nil case logical.LocalRelation(output, data, _, stream) => LocalTableScanExec(output, data, stream) :: Nil diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala index 0d34dfc91c39f..e31e81fc1fa90 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala @@ -22,13 +22,14 @@ import scala.collection.mutable import org.apache.spark.internal.{Logging, LogKeys} import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.expressions.{AttributeReference, AttributeSet, DynamicPruning, DynamicPruningExpression, Expression, ExpressionSet, GetStructField, NamedExpression, PythonUDF, SchemaPruning, SubqueryExpression, V2ExpressionUtils} +import org.apache.spark.sql.catalyst.plans.logical.SampleMethod import org.apache.spark.sql.catalyst.types.DataTypeUtils import org.apache.spark.sql.catalyst.types.DataTypeUtils.toAttributes import org.apache.spark.sql.catalyst.util.CharVarcharUtils import org.apache.spark.sql.connector.catalog.Table import org.apache.spark.sql.connector.expressions.{IdentityTransform, SortOrder} import org.apache.spark.sql.connector.expressions.filter.Predicate -import org.apache.spark.sql.connector.read.{Scan, ScanBuilder, SupportsPushDownFilters, SupportsPushDownLimit, SupportsPushDownOffset, SupportsPushDownRequiredColumns, SupportsPushDownTableSample, SupportsPushDownTopN, SupportsPushDownV2Filters, SupportsRuntimeV2Filtering} +import org.apache.spark.sql.connector.read.{SampleMethod => SampleMethodV2, Scan, ScanBuilder, SupportsPushDownFilters, SupportsPushDownLimit, SupportsPushDownOffset, SupportsPushDownRequiredColumns, SupportsPushDownTableSample, SupportsPushDownTopN, SupportsPushDownV2Filters, SupportsRuntimeV2Filtering} import org.apache.spark.sql.execution.{ScalarSubquery => ExecScalarSubquery} import org.apache.spark.sql.execution.datasources.{DataSourceStrategy, DataSourceUtils} import org.apache.spark.sql.internal.SQLConf @@ -398,7 +399,11 @@ object PushDownUtils extends Logging { scanBuilder match { case s: SupportsPushDownTableSample => s.pushTableSample( - sample.lowerBound, sample.upperBound, sample.withReplacement, sample.seed) + sample.lowerBound, sample.upperBound, sample.withReplacement, sample.seed, + sample.sampleMethod match { + case SampleMethod.Bernoulli => SampleMethodV2.BERNOULLI + case SampleMethod.System => SampleMethodV2.SYSTEM + }) case _ => false } } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableSampleInfo.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableSampleInfo.scala index cb4fb9eb0809a..441ed28c813c0 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableSampleInfo.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableSampleInfo.scala @@ -17,8 +17,11 @@ package org.apache.spark.sql.execution.datasources.v2 +import org.apache.spark.sql.catalyst.plans.logical.SampleMethod + case class TableSampleInfo( lowerBound: Double, upperBound: Double, withReplacement: Boolean, - seed: Long) + seed: Long, + sampleMethod: SampleMethod = SampleMethod.Bernoulli) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala index c0b72123065f7..60a2017e6947c 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala @@ -23,11 +23,12 @@ import scala.collection.mutable import org.apache.spark.{SparkException, SparkIllegalArgumentException} import org.apache.spark.internal.LogKeys.{AGGREGATE_FUNCTIONS, COLUMN_NAMES, GROUP_BY_EXPRS, JOIN_CONDITION, JOIN_TYPE, POST_SCAN_FILTERS, PUSHED_FILTERS, RELATION_NAME, RELATION_OUTPUT} +import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.expressions.{aggregate, Alias, And, Attribute, AttributeMap, AttributeReference, AttributeSet, Cast, Expression, ExpressionSet, ExprId, IntegerLiteral, Literal, NamedExpression, PredicateHelper, ProjectionOverSchema, SortOrder, SubqueryExpression} import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression import org.apache.spark.sql.catalyst.optimizer.CollapseProject import org.apache.spark.sql.catalyst.planning.{PhysicalOperation, ScanOperation} -import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Join, LeafNode, Limit, LimitAndOffset, LocalLimit, LogicalPlan, Offset, OffsetAndLimit, Project, Sample, Sort} +import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Join, LeafNode, Limit, LimitAndOffset, LocalLimit, LogicalPlan, Offset, OffsetAndLimit, Project, Sample, SampleMethod, Sort} import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.catalyst.types.DataTypeUtils.toAttributes import org.apache.spark.sql.connector.expressions.{SortOrder => V2SortOrder} @@ -150,6 +151,11 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper { rightProjections.forall(_.isInstanceOf[AttributeReference]) && // Cross joins are not supported because they increase the amount of data. condition.isDefined && + // Do not push down join if either side has a pushed sample, because + // the merged scan builder would silently discard it. + // TODO(SPARK-56504): Extend SupportsPushDownJoin to accept pushed + // samples so sources supporting both can handle the composition. + leftHolder.pushedSample.isEmpty && rightHolder.pushedSample.isEmpty && lBuilder.isOtherSideCompatibleForJoin(rBuilder) => // Process left and right columns in original order val (leftSideRequiredColumnsWithAliases, rightSideRequiredColumnsWithAliases) = @@ -844,15 +850,26 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper { sample.lowerBound, sample.upperBound, sample.withReplacement, - sample.seed.getOrElse((math.random() * 1000).toLong)) + // TODO(SPARK-56573): The * 1000 limits the seed to only 1000 distinct values. + // Kept here for consistency with SampleExec.resolvedSeed; will be fixed + // across all call sites in SPARK-56573. + sample.seed.getOrElse((math.random() * 1000).toLong), + sampleMethod = sample.sampleMethod) val pushed = PushDownUtils.pushTableSample(sHolder.builder, tableSample) if (pushed) { sHolder.pushedSample = Some(tableSample) sample.child + } else if (sample.sampleMethod == SampleMethod.System) { + throw new AnalysisException( + errorClass = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM", + messageParameters = Map.empty) } else { sample } - + case _ if sample.sampleMethod == SampleMethod.System => + throw new AnalysisException( + errorClass = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_NO_SCAN", + messageParameters = Map.empty) case _ => sample } } diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/pipe-operators.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/pipe-operators.sql.out index 84ec13334ffd0..a6a86f9ebe1dd 100644 --- a/sql/core/src/test/resources/sql-tests/analyzer-results/pipe-operators.sql.out +++ b/sql/core/src/test/resources/sql-tests/analyzer-results/pipe-operators.sql.out @@ -1979,7 +1979,7 @@ org.apache.spark.sql.catalyst.parser.ParseException table t |> tablesample (100 percent) repeatable (0) -- !query analysis -Sample 0.0, 1.0, false, 0 +Sample 0.0, 1.0, false, 0, Bernoulli +- SubqueryAlias spark_catalog.default.t +- Relation spark_catalog.default.t[x#x,y#x] csv @@ -1998,7 +1998,7 @@ GlobalLimit 2 table t |> tablesample (bucket 1 out of 1) repeatable (0) -- !query analysis -Sample 0.0, 1.0, false, 0 +Sample 0.0, 1.0, false, 0, Bernoulli +- SubqueryAlias spark_catalog.default.t +- Relation spark_catalog.default.t[x#x,y#x] csv @@ -2009,10 +2009,10 @@ table t |> tablesample (5 rows) repeatable (0) |> tablesample (bucket 1 out of 1) repeatable (0) -- !query analysis -Sample 0.0, 1.0, false, 0 +Sample 0.0, 1.0, false, 0, Bernoulli +- GlobalLimit 5 +- LocalLimit 5 - +- Sample 0.0, 1.0, false, 0 + +- Sample 0.0, 1.0, false, 0, Bernoulli +- SubqueryAlias spark_catalog.default.t +- Relation spark_catalog.default.t[x#x,y#x] csv @@ -2435,7 +2435,7 @@ Project [a#x] : +- Project [a#x] : +- SubqueryAlias grouping : +- LocalRelation [a#x] - +- Sample 0.0, 1.0, false, 0 + +- Sample 0.0, 1.0, false, 0, Bernoulli +- SubqueryAlias jt2 +- SubqueryAlias join_test_t2 +- View (`join_test_t2`, [a#x]) @@ -2458,7 +2458,7 @@ Project [a#x] : +- SubqueryAlias grouping : +- LocalRelation [a#x] +- SubqueryAlias jt2 - +- Sample 0.0, 1.0, false, 0 + +- Sample 0.0, 1.0, false, 0, Bernoulli +- Project [1 AS a#x] +- OneRowRelation diff --git a/sql/core/src/test/resources/sql-tests/results/keywords-enforced.sql.out b/sql/core/src/test/resources/sql-tests/results/keywords-enforced.sql.out index 6f9e8fde5d9f1..6bcbdd2840f90 100644 --- a/sql/core/src/test/resources/sql-tests/results/keywords-enforced.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/keywords-enforced.sql.out @@ -25,6 +25,7 @@ AT false ATOMIC false AUTHORIZATION true BEGIN false +BERNOULLI false BETWEEN false BIGINT false BINARY false @@ -358,6 +359,7 @@ STRUCT false SUBSTR false SUBSTRING false SYNC false +SYSTEM false SYSTEM_PATH false SYSTEM_TIME false SYSTEM_VERSION false diff --git a/sql/core/src/test/resources/sql-tests/results/keywords.sql.out b/sql/core/src/test/resources/sql-tests/results/keywords.sql.out index 1fdb51507bc1b..a010343264469 100644 --- a/sql/core/src/test/resources/sql-tests/results/keywords.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/keywords.sql.out @@ -25,6 +25,7 @@ AT false ATOMIC false AUTHORIZATION false BEGIN false +BERNOULLI false BETWEEN false BIGINT false BINARY false @@ -358,6 +359,7 @@ STRUCT false SUBSTR false SUBSTRING false SYNC false +SYSTEM false SYSTEM_PATH false SYSTEM_TIME false SYSTEM_VERSION false diff --git a/sql/core/src/test/resources/sql-tests/results/nonansi/keywords.sql.out b/sql/core/src/test/resources/sql-tests/results/nonansi/keywords.sql.out index 1fdb51507bc1b..a010343264469 100644 --- a/sql/core/src/test/resources/sql-tests/results/nonansi/keywords.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/nonansi/keywords.sql.out @@ -25,6 +25,7 @@ AT false ATOMIC false AUTHORIZATION false BEGIN false +BERNOULLI false BETWEEN false BIGINT false BINARY false @@ -358,6 +359,7 @@ STRUCT false SUBSTR false SUBSTRING false SYNC false +SYSTEM false SYSTEM_PATH false SYSTEM_TIME false SYSTEM_VERSION false diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2TableSampleSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2TableSampleSuite.scala new file mode 100644 index 0000000000000..76ec2e588eae6 --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2TableSampleSuite.scala @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector + +import org.apache.spark.sql.AnalysisException +import org.apache.spark.sql.connector.catalog.{InMemoryTableWithJoinAndSampleCatalog, InMemoryTableWithLegacyTableSampleCatalog, InMemoryTableWithTableSampleCatalog} +import org.apache.spark.sql.internal.SQLConf + +class DataSourceV2TableSampleSuite extends DatasourceV2SQLBase + with DataSourcePushdownTestUtils { + + private val sampleCatalog = "testsample" + + private def withSampleTable(testFunc: String => Unit): Unit = { + registerCatalog(sampleCatalog, classOf[InMemoryTableWithTableSampleCatalog]) + val tableName = s"$sampleCatalog.ns.sample_tbl" + sql(s"CREATE TABLE $tableName (id bigint, data string) USING _") + try { + sql(s"INSERT INTO $tableName VALUES (1, 'a'), (2, 'b'), (3, 'c'), (4, 'd'), (5, 'e')") + testFunc(tableName) + } finally { + sql(s"DROP TABLE IF EXISTS $tableName") + } + } + + test("SPARK-55978: TABLESAMPLE SYSTEM pushdown removes Sample node") { + withSampleTable { table => + val df = sql(s"SELECT * FROM $table TABLESAMPLE SYSTEM (50 PERCENT)") + checkSamplePushed(df, pushed = true) + checkPushedInfo(df, "SYSTEM SAMPLE (50.0) false SEED(") + } + } + + test("SPARK-55978: TABLESAMPLE BERNOULLI pushdown removes Sample node") { + withSampleTable { table => + val df = sql(s"SELECT * FROM $table TABLESAMPLE BERNOULLI (50 PERCENT)") + checkSamplePushed(df, pushed = true) + checkPushedInfo(df, "BERNOULLI SAMPLE (50.0) false SEED(") + } + } + + test("SPARK-55978: TABLESAMPLE default (no qualifier) pushdown removes Sample node") { + withSampleTable { table => + val df = sql(s"SELECT * FROM $table TABLESAMPLE (50 PERCENT)") + checkSamplePushed(df, pushed = true) + } + } + + test("SPARK-55978: TABLESAMPLE SYSTEM 0 PERCENT returns no rows") { + withSampleTable { table => + val df = sql(s"SELECT * FROM $table TABLESAMPLE SYSTEM (0 PERCENT)") + checkSamplePushed(df, pushed = true) + assert(df.collect().isEmpty) + } + } + + test("SPARK-55978: TABLESAMPLE SYSTEM 100 PERCENT returns all rows") { + withSampleTable { table => + val df = sql(s"SELECT * FROM $table TABLESAMPLE SYSTEM (100 PERCENT)") + checkSamplePushed(df, pushed = true) + assert(df.collect().length == 5) + } + } + + test("SPARK-55978: TABLESAMPLE SYSTEM composes with projection") { + withSampleTable { table => + val df = sql(s"SELECT id FROM $table TABLESAMPLE SYSTEM (100 PERCENT)") + checkSamplePushed(df, pushed = true) + assert(df.columns.sameElements(Array("id"))) + assert(df.collect().length == 5) + } + } + + test("SPARK-55978: TABLESAMPLE on non-pushdown catalog falls back to Sample node") { + val table = "testcat.ns.no_sample_tbl" + sql(s"CREATE TABLE $table (id bigint, data string) USING _") + try { + sql(s"INSERT INTO $table VALUES (1, 'a'), (2, 'b'), (3, 'c')") + val df = sql(s"SELECT * FROM $table TABLESAMPLE (50 PERCENT)") + // testcat uses InMemoryCatalog which does NOT implement SupportsPushDownTableSample, + // so the Sample node should remain in the plan. + checkSamplePushed(df, pushed = false) + } finally { + sql(s"DROP TABLE IF EXISTS $table") + } + } + + test("SPARK-55978: TABLESAMPLE SYSTEM on non-pushdown catalog errors") { + val table = "testcat.ns.no_sample_tbl" + sql(s"CREATE TABLE $table (id bigint, data string) USING _") + try { + sql(s"INSERT INTO $table VALUES (1, 'a'), (2, 'b'), (3, 'c')") + // testcat uses InMemoryCatalog whose ScanBuilder does not implement + // SupportsPushDownTableSample, so SYSTEM sampling cannot be pushed down. + checkError( + exception = intercept[AnalysisException] { + sql(s"SELECT * FROM $table TABLESAMPLE SYSTEM (50 PERCENT)").collect() + }, + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM") + } finally { + sql(s"DROP TABLE IF EXISTS $table") + } + } + + test("SPARK-55978: TABLESAMPLE SYSTEM on subquery errors") { + withSampleTable { table => + // SYSTEM sampling requires a direct table scan; applying it to a derived + // query (here an aggregate) means there is no ScanBuilderHolder to push into. + checkError( + exception = intercept[AnalysisException] { + sql(s"SELECT * FROM (SELECT id, count(*) AS cnt FROM $table GROUP BY id) " + + s"TABLESAMPLE SYSTEM (50 PERCENT)").collect() + }, + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_NO_SCAN") + } + } + + test("SPARK-55978: TABLESAMPLE SYSTEM with WHERE filter errors") { + withSampleTable { table => + // A WHERE clause between the Sample and the scan produces a non-empty filter list + // in PhysicalOperation, which falls through to the catch-all error branch. + checkError( + exception = intercept[AnalysisException] { + sql(s"SELECT * FROM (SELECT * FROM $table WHERE id > 1) " + + s"TABLESAMPLE SYSTEM (50 PERCENT)").collect() + }, + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_NO_SCAN") + } + } + + test("SPARK-55978: TABLESAMPLE SYSTEM on DSv1 table errors") { + withTable("dsv1_tbl") { + sql("CREATE TABLE dsv1_tbl (id bigint, data string) USING parquet") + sql("INSERT INTO dsv1_tbl VALUES (1, 'a'), (2, 'b'), (3, 'c')") + // DSv1 tables have no ScanBuilderHolder, so SYSTEM sampling cannot be pushed down. + checkError( + exception = intercept[AnalysisException] { + sql("SELECT * FROM dsv1_tbl TABLESAMPLE SYSTEM (50 PERCENT)").collect() + }, + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM_NO_SCAN") + } + } + + test("SPARK-55978: join pushdown is skipped when a side has a pushed sample") { + val joinSampleCatalog = "testjoinsample" + registerCatalog(joinSampleCatalog, classOf[InMemoryTableWithJoinAndSampleCatalog]) + val t1 = s"$joinSampleCatalog.ns.t1" + val t2 = s"$joinSampleCatalog.ns.t2" + sql(s"CREATE TABLE $t1 (id bigint, data string) USING _") + sql(s"CREATE TABLE $t2 (id bigint, data string) USING _") + try { + sql(s"INSERT INTO $t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')") + sql(s"INSERT INTO $t2 VALUES (2, 'x'), (3, 'y'), (4, 'z')") + withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") { + // Without sample: join should be pushed down + val dfNoSample = sql(s"SELECT * FROM $t1 JOIN $t2 ON $t1.id = $t2.id") + checkJoinPushed(dfNoSample) + + // With SYSTEM sample on one side: join pushdown should be skipped + val dfWithSample = sql( + s"SELECT * FROM $t1 TABLESAMPLE SYSTEM (100 PERCENT) " + + s"JOIN $t2 ON $t1.id = $t2.id") + checkJoinNotPushed(dfWithSample) + // The sample should still be pushed down though + checkSamplePushed(dfWithSample, pushed = true) + } + } finally { + sql(s"DROP TABLE IF EXISTS $t1") + sql(s"DROP TABLE IF EXISTS $t2") + } + } + + test("SPARK-55978: legacy connector with only 4-arg pushTableSample - BERNOULLI pushes down") { + val legacyCatalog = "testlegacysample" + registerCatalog(legacyCatalog, classOf[InMemoryTableWithLegacyTableSampleCatalog]) + val tableName = s"$legacyCatalog.ns.legacy_tbl" + sql(s"CREATE TABLE $tableName (id bigint, data string) USING _") + try { + sql(s"INSERT INTO $tableName VALUES (1, 'a'), (2, 'b'), (3, 'c')") + // BERNOULLI should push down via the default 5-arg method delegating to 4-arg + val dfBernoulli = sql(s"SELECT * FROM $tableName TABLESAMPLE (50 PERCENT)") + checkSamplePushed(dfBernoulli, pushed = true) + + // SYSTEM should fail because the default 5-arg method returns false for SYSTEM, + // and SYSTEM requires successful pushdown. + checkError( + exception = intercept[AnalysisException] { + sql(s"SELECT * FROM $tableName TABLESAMPLE SYSTEM (50 PERCENT)").collect() + }, + condition = "UNSUPPORTED_FEATURE.TABLESAMPLE_SYSTEM") + } finally { + sql(s"DROP TABLE IF EXISTS $tableName") + } + } +} diff --git a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerWithSparkContextSuite.scala b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerWithSparkContextSuite.scala index 5067f7dfbcc54..1ecf5b3dae4a0 100644 --- a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerWithSparkContextSuite.scala +++ b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerWithSparkContextSuite.scala @@ -214,7 +214,7 @@ trait ThriftServerWithSparkContextSuite extends SharedThriftServer { val sessionHandle = client.openSession(user, "") val infoValue = client.getInfo(sessionHandle, GetInfoType.CLI_ODBC_KEYWORDS) // scalastyle:off line.size.limit - assert(infoValue.getStringValue == "ADD,AFTER,AGGREGATE,ALL,ALTER,ALWAYS,ANALYZE,AND,ANTI,ANY,ANY_VALUE,APPROX,ARCHIVE,ARRAY,AS,ASC,ASENSITIVE,AT,ATOMIC,AUTHORIZATION,BEGIN,BETWEEN,BIGINT,BINARY,BINDING,BOOLEAN,BOTH,BUCKET,BUCKETS,BY,BYTE,CACHE,CALL,CALLED,CASCADE,CASE,CAST,CATALOG,CATALOGS,CHANGE,CHANGES,CHAR,CHARACTER,CHECK,CLEAR,CLOSE,CLUSTER,CLUSTERED,CODEGEN,COLLATE,COLLATION,COLLATIONS,COLLECTION,COLUMN,COLUMNS,COMMENT,COMMIT,COMPACT,COMPACTIONS,COMPENSATION,COMPUTE,CONCATENATE,CONDITION,CONSTRAINT,CONTAINS,CONTINUE,COST,CREATE,CROSS,CUBE,CURRENT,CURRENT_DATABASE,CURRENT_DATE,CURRENT_PATH,CURRENT_SCHEMA,CURRENT_TIME,CURRENT_TIMESTAMP,CURRENT_USER,CURSOR,DATA,DATABASE,DATABASES,DATE,DATEADD,DATEDIFF,DATE_ADD,DATE_DIFF,DAY,DAYOFYEAR,DAYS,DBPROPERTIES,DEC,DECIMAL,DECLARE,DEFAULT,DEFAULT_PATH,DEFINED,DEFINER,DELAY,DELETE,DELIMITED,DESC,DESCRIBE,DETERMINISTIC,DFS,DIRECTORIES,DIRECTORY,DISTANCE,DISTINCT,DISTRIBUTE,DIV,DO,DOUBLE,DROP,ELSE,ELSEIF,END,ENFORCED,ESCAPE,ESCAPED,EVOLUTION,EXACT,EXCEPT,EXCHANGE,EXCLUDE,EXCLUSIVE,EXECUTE,EXISTS,EXIT,EXPLAIN,EXPORT,EXTEND,EXTENDED,EXTERNAL,EXTRACT,FALSE,FETCH,FIELDS,FILEFORMAT,FILTER,FIRST,FLOAT,FLOW,FOLLOWING,FOR,FOREIGN,FORMAT,FORMATTED,FOUND,FROM,FULL,FUNCTION,FUNCTIONS,GENERATED,GEOGRAPHY,GEOMETRY,GLOBAL,GRANT,GROUP,GROUPING,HANDLER,HAVING,HOUR,HOURS,IDENTIFIED,IDENTIFIER,IDENTITY,IF,IGNORE,ILIKE,IMMEDIATE,IMPORT,IN,INCLUDE,INCLUSIVE,INCREMENT,INDEX,INDEXES,INNER,INPATH,INPUT,INPUTFORMAT,INSENSITIVE,INSERT,INT,INTEGER,INTERSECT,INTERVAL,INTO,INVOKER,IS,ITEMS,ITERATE,JOIN,JSON,KEY,KEYS,LANGUAGE,LAST,LATERAL,LAZY,LEADING,LEAVE,LEFT,LEVEL,LIKE,LIMIT,LINES,LIST,LOAD,LOCAL,LOCATION,LOCK,LOCKS,LOGICAL,LONG,LOOP,MACRO,MAP,MATCHED,MATERIALIZED,MAX,MEASURE,MERGE,METRICS,MICROSECOND,MICROSECONDS,MILLISECOND,MILLISECONDS,MINUS,MINUTE,MINUTES,MODIFIES,MONTH,MONTHS,MSCK,NAME,NAMESPACE,NAMESPACES,NANOSECOND,NANOSECONDS,NATURAL,NEAREST,NEXT,NO,NONE,NORELY,NOT,NULL,NULLS,NUMERIC,OF,OFFSET,ON,ONLY,OPEN,OPTION,OPTIONS,OR,ORDER,OUT,OUTER,OUTPUTFORMAT,OVER,OVERLAPS,OVERLAY,OVERWRITE,PARTITION,PARTITIONED,PARTITIONS,PATH,PERCENT,PIVOT,PLACING,POSITION,PRECEDING,PRIMARY,PRINCIPALS,PROCEDURE,PROCEDURES,PROPERTIES,PURGE,QUALIFY,QUARTER,QUERY,RANGE,READ,READS,REAL,RECORDREADER,RECORDWRITER,RECOVER,RECURSION,RECURSIVE,REDUCE,REFERENCES,REFRESH,RELY,RENAME,REPAIR,REPEAT,REPEATABLE,REPLACE,RESET,RESPECT,RESTRICT,RETURN,RETURNS,REVOKE,RIGHT,ROLE,ROLES,ROLLBACK,ROLLUP,ROW,ROWS,SCHEMA,SCHEMAS,SECOND,SECONDS,SECURITY,SELECT,SEMI,SEPARATED,SERDE,SERDEPROPERTIES,SESSION_USER,SET,SETS,SHORT,SHOW,SIMILARITY,SINGLE,SKEWED,SMALLINT,SOME,SORT,SORTED,SOURCE,SPECIFIC,SQL,SQLEXCEPTION,SQLSTATE,START,STATISTICS,STORED,STRATIFY,STREAM,STREAMING,STRING,STRUCT,SUBSTR,SUBSTRING,SYNC,SYSTEM_PATH,SYSTEM_TIME,SYSTEM_VERSION,TABLE,TABLES,TABLESAMPLE,TARGET,TBLPROPERTIES,TERMINATED,THEN,TIME,TIMEDIFF,TIMESTAMP,TIMESTAMPADD,TIMESTAMPDIFF,TIMESTAMP_LTZ,TIMESTAMP_NTZ,TINYINT,TO,TOUCH,TRAILING,TRANSACTION,TRANSACTIONS,TRANSFORM,TRIM,TRUE,TRUNCATE,TRY_CAST,TYPE,UNARCHIVE,UNBOUNDED,UNCACHE,UNION,UNIQUE,UNKNOWN,UNLOCK,UNPIVOT,UNSET,UNTIL,UPDATE,USE,USER,USING,VALUE,VALUES,VAR,VARCHAR,VARIABLE,VARIANT,VERSION,VIEW,VIEWS,VOID,WATERMARK,WEEK,WEEKS,WHEN,WHERE,WHILE,WINDOW,WITH,WITHIN,WITHOUT,X,YEAR,YEARS,ZONE") + assert(infoValue.getStringValue == "ADD,AFTER,AGGREGATE,ALL,ALTER,ALWAYS,ANALYZE,AND,ANTI,ANY,ANY_VALUE,APPROX,ARCHIVE,ARRAY,AS,ASC,ASENSITIVE,AT,ATOMIC,AUTHORIZATION,BEGIN,BERNOULLI,BETWEEN,BIGINT,BINARY,BINDING,BOOLEAN,BOTH,BUCKET,BUCKETS,BY,BYTE,CACHE,CALL,CALLED,CASCADE,CASE,CAST,CATALOG,CATALOGS,CHANGE,CHANGES,CHAR,CHARACTER,CHECK,CLEAR,CLOSE,CLUSTER,CLUSTERED,CODEGEN,COLLATE,COLLATION,COLLATIONS,COLLECTION,COLUMN,COLUMNS,COMMENT,COMMIT,COMPACT,COMPACTIONS,COMPENSATION,COMPUTE,CONCATENATE,CONDITION,CONSTRAINT,CONTAINS,CONTINUE,COST,CREATE,CROSS,CUBE,CURRENT,CURRENT_DATABASE,CURRENT_DATE,CURRENT_PATH,CURRENT_SCHEMA,CURRENT_TIME,CURRENT_TIMESTAMP,CURRENT_USER,CURSOR,DATA,DATABASE,DATABASES,DATE,DATEADD,DATEDIFF,DATE_ADD,DATE_DIFF,DAY,DAYOFYEAR,DAYS,DBPROPERTIES,DEC,DECIMAL,DECLARE,DEFAULT,DEFAULT_PATH,DEFINED,DEFINER,DELAY,DELETE,DELIMITED,DESC,DESCRIBE,DETERMINISTIC,DFS,DIRECTORIES,DIRECTORY,DISTANCE,DISTINCT,DISTRIBUTE,DIV,DO,DOUBLE,DROP,ELSE,ELSEIF,END,ENFORCED,ESCAPE,ESCAPED,EVOLUTION,EXACT,EXCEPT,EXCHANGE,EXCLUDE,EXCLUSIVE,EXECUTE,EXISTS,EXIT,EXPLAIN,EXPORT,EXTEND,EXTENDED,EXTERNAL,EXTRACT,FALSE,FETCH,FIELDS,FILEFORMAT,FILTER,FIRST,FLOAT,FLOW,FOLLOWING,FOR,FOREIGN,FORMAT,FORMATTED,FOUND,FROM,FULL,FUNCTION,FUNCTIONS,GENERATED,GEOGRAPHY,GEOMETRY,GLOBAL,GRANT,GROUP,GROUPING,HANDLER,HAVING,HOUR,HOURS,IDENTIFIED,IDENTIFIER,IDENTITY,IF,IGNORE,ILIKE,IMMEDIATE,IMPORT,IN,INCLUDE,INCLUSIVE,INCREMENT,INDEX,INDEXES,INNER,INPATH,INPUT,INPUTFORMAT,INSENSITIVE,INSERT,INT,INTEGER,INTERSECT,INTERVAL,INTO,INVOKER,IS,ITEMS,ITERATE,JOIN,JSON,KEY,KEYS,LANGUAGE,LAST,LATERAL,LAZY,LEADING,LEAVE,LEFT,LEVEL,LIKE,LIMIT,LINES,LIST,LOAD,LOCAL,LOCATION,LOCK,LOCKS,LOGICAL,LONG,LOOP,MACRO,MAP,MATCHED,MATERIALIZED,MAX,MEASURE,MERGE,METRICS,MICROSECOND,MICROSECONDS,MILLISECOND,MILLISECONDS,MINUS,MINUTE,MINUTES,MODIFIES,MONTH,MONTHS,MSCK,NAME,NAMESPACE,NAMESPACES,NANOSECOND,NANOSECONDS,NATURAL,NEAREST,NEXT,NO,NONE,NORELY,NOT,NULL,NULLS,NUMERIC,OF,OFFSET,ON,ONLY,OPEN,OPTION,OPTIONS,OR,ORDER,OUT,OUTER,OUTPUTFORMAT,OVER,OVERLAPS,OVERLAY,OVERWRITE,PARTITION,PARTITIONED,PARTITIONS,PATH,PERCENT,PIVOT,PLACING,POSITION,PRECEDING,PRIMARY,PRINCIPALS,PROCEDURE,PROCEDURES,PROPERTIES,PURGE,QUALIFY,QUARTER,QUERY,RANGE,READ,READS,REAL,RECORDREADER,RECORDWRITER,RECOVER,RECURSION,RECURSIVE,REDUCE,REFERENCES,REFRESH,RELY,RENAME,REPAIR,REPEAT,REPEATABLE,REPLACE,RESET,RESPECT,RESTRICT,RETURN,RETURNS,REVOKE,RIGHT,ROLE,ROLES,ROLLBACK,ROLLUP,ROW,ROWS,SCHEMA,SCHEMAS,SECOND,SECONDS,SECURITY,SELECT,SEMI,SEPARATED,SERDE,SERDEPROPERTIES,SESSION_USER,SET,SETS,SHORT,SHOW,SIMILARITY,SINGLE,SKEWED,SMALLINT,SOME,SORT,SORTED,SOURCE,SPECIFIC,SQL,SQLEXCEPTION,SQLSTATE,START,STATISTICS,STORED,STRATIFY,STREAM,STREAMING,STRING,STRUCT,SUBSTR,SUBSTRING,SYNC,SYSTEM,SYSTEM_PATH,SYSTEM_TIME,SYSTEM_VERSION,TABLE,TABLES,TABLESAMPLE,TARGET,TBLPROPERTIES,TERMINATED,THEN,TIME,TIMEDIFF,TIMESTAMP,TIMESTAMPADD,TIMESTAMPDIFF,TIMESTAMP_LTZ,TIMESTAMP_NTZ,TINYINT,TO,TOUCH,TRAILING,TRANSACTION,TRANSACTIONS,TRANSFORM,TRIM,TRUE,TRUNCATE,TRY_CAST,TYPE,UNARCHIVE,UNBOUNDED,UNCACHE,UNION,UNIQUE,UNKNOWN,UNLOCK,UNPIVOT,UNSET,UNTIL,UPDATE,USE,USER,USING,VALUE,VALUES,VAR,VARCHAR,VARIABLE,VARIANT,VERSION,VIEW,VIEWS,VOID,WATERMARK,WEEK,WEEKS,WHEN,WHERE,WHILE,WINDOW,WITH,WITHIN,WITHOUT,X,YEAR,YEARS,ZONE") // scalastyle:on line.size.limit } } From 49af916d4e41bc0bdbb3198e90b6889e28872cf5 Mon Sep 17 00:00:00 2001 From: Kousuke Saruta Date: Tue, 12 May 2026 12:40:00 +0900 Subject: [PATCH 102/286] [SPARK-56812][INFRA] Fix URL of get-pip.py in dev/infra/Dockerfile for Python 3.9 ### What changes were proposed in this pull request? This PR fixes a URL of `get-pip.py` in `dev/infra/Dockerfile` for Python 3.9. ### Why are the changes needed? Currently `docker build dev/infra` fails with following error. ``` => ERROR [15/32] RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9 0.6s ------ > [15/32] RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9: 0.588 ERROR: This script does not work on Python 3.9. The minimum supported Python version is 3.10. Please use https://bootstrap.pypa.io/pip/3.9/get-pip.py instead. ------ Dockerfile:111 -------------------- 109 | python3.9 python3.9-distutils \ 110 | && rm -rf /var/lib/apt/lists/* 111 | >>> RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9 112 | RUN python3.9 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow needs this 113 | RUN python3.9 -m pip install --force $BASIC_PIP_PKGS unittest-xml-reporting $CONNECT_PIP_PKGS && \ -------------------- ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Confirmed `docker build dev/infra` works. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55788 from sarutak/fix-get-pip-url. Authored-by: Kousuke Saruta Signed-off-by: Kousuke Saruta (cherry picked from commit 3eec6b6fb4d4779ea98418755131138f315854aa) Signed-off-by: Kousuke Saruta --- dev/infra/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile index 1cfc22acc2302..b848f8eb754da 100644 --- a/dev/infra/Dockerfile +++ b/dev/infra/Dockerfile @@ -108,7 +108,7 @@ RUN add-apt-repository ppa:deadsnakes/ppa RUN apt-get update && apt-get install -y \ python3.9 python3.9-distutils \ && rm -rf /var/lib/apt/lists/* -RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9 +RUN curl -sS https://bootstrap.pypa.io/pip/3.9/get-pip.py | python3.9 RUN python3.9 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow needs this RUN python3.9 -m pip install --force $BASIC_PIP_PKGS unittest-xml-reporting $CONNECT_PIP_PKGS && \ python3.9 -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu && \ From 8dffe020266e073ecd5d3206b5d299005ba7da25 Mon Sep 17 00:00:00 2001 From: Ruifeng Zheng Date: Tue, 12 May 2026 13:46:11 +0800 Subject: [PATCH 103/286] [SPARK-56831][INFRA][R] Share SBT precompile artifact with sparkr CI job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? Follow-up to [SPARK-56768](https://issues.apache.org/jira/browse/SPARK-56768) (apache/spark#55726), which introduced a shared `precompile` CI job that runs Spark's SBT build once and publishes the resulting `target/` trees as a GitHub Actions artifact for the pyspark matrix entries to consume. This PR extends that same artifact to the `sparkr` build. Concretely: - The `precompile` job's `if:` gate now also fires when `sparkr == 'true'` is set in the precondition output, so the artifact is built whenever only sparkr changes. - The `sparkr` job adds `precompile` to `needs:`, downloads and extracts the artifact (with the same graceful fallback as the pyspark matrix), and exports `SKIP_SCALA_BUILD=true` for `dev/run-tests.py` only when the artifact was successfully extracted. - No `dev/run-tests.py` change is needed — the `SKIP_SCALA_BUILD` gate landed with SPARK-56768. ### Optional: graceful fallback if precompile fails Same pattern as the pyspark matrix: - The "Download precompiled artifact" step is gated on `needs.precompile.result == 'success'` and has `continue-on-error: true`. - The "Extract precompiled artifact" step is gated on the download succeeding and also has `continue-on-error: true`. - Inside the "Run tests" bash block, `SKIP_SCALA_BUILD=true` is exported only when `steps.extract-precompiled.outcome == 'success'`. Otherwise it stays unset and `dev/run-tests.py` falls back to the original local SBT build. So a precompile/download/extract failure degrades sparkr to the pre-PR behavior, not a workflow failure. ### Why are the changes needed? The sparkr job today runs the same ~13m of redundant SBT compile that the pyspark matrix used to run. Reusing the existing precompile artifact removes that redundant work. The `precompile` job is already running in any workflow run where pyspark changes are present; adding sparkr as another consumer is essentially free (just another download of the same artifact). When sparkr is the only changed module, the `precompile` job is now scheduled to run anyway (via the new `sparkr == 'true'` clause in its `if:` gate), so this case picks up the same saving. ### Estimated savings | | Per sparkr run | |---|---:| | Redundant SBT compile in sparkr today | ~13m | | Add back: download + extract overhead | ~1m | | **Net CI compute saved per sparkr run** | **~12m** | This is on top of the ~96m / ~14% already saved by SPARK-56768. The actual wall clock for the sparkr job will drop by roughly the same amount (sparkr is not on the critical path; the pyspark matrix still drives the workflow's wall-clock). ### Does this PR introduce _any_ user-facing change? No. CI infrastructure change only. ### How was this patch tested? The change is exercised by the CI run of this PR itself, when the sparkr job runs. The expected log signature inside "Run tests" is `Reusing precompiled artifact, skipping local SBT build.`, mirroring what the pyspark matrix already prints. If the precompile artifact is not available (precompile job failed, or this is some future caller that doesn't enable it), sparkr falls back to the local SBT build path, which is identical to today's behavior. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes #55761 from zhengruifeng/share-precompile-sparkr. Authored-by: Ruifeng Zheng Signed-off-by: Ruifeng Zheng (cherry picked from commit ef4e78489f4a2fc2635e96170934ee5791534588) Signed-off-by: Ruifeng Zheng --- .github/workflows/build_and_test.yml | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 99223efbf0dd2..3f3400153cde7 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -540,7 +540,8 @@ jobs: (!cancelled()) && ( fromJson(needs.precondition.outputs.required).pyspark == 'true' || fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true' || - fromJson(needs.precondition.outputs.required).pyspark-install == 'true') + fromJson(needs.precondition.outputs.required).pyspark-install == 'true' || + fromJson(needs.precondition.outputs.required).sparkr == 'true') name: "Precompile Spark" runs-on: ubuntu-latest timeout-minutes: 60 @@ -806,7 +807,7 @@ jobs: path: "**/target/unit-tests.log" sparkr: - needs: [precondition, infra-image] + needs: [precondition, infra-image, precompile] # always run if sparkr == 'true', even infra-image is skip (such as non-master job) if: (!cancelled()) && fromJson(needs.precondition.outputs.required).sparkr == 'true' name: "Build modules: sparkr" @@ -865,6 +866,20 @@ jobs: with: distribution: zulu java-version: ${{ inputs.java }} + - name: Download precompiled artifact + id: download-precompiled + if: needs.precompile.result == 'success' + continue-on-error: true + uses: actions/download-artifact@v6 + with: + name: spark-compile-${{ inputs.branch }}-${{ github.run_id }} + - name: Extract precompiled artifact + id: extract-precompiled + if: steps.download-precompiled.outcome == 'success' + continue-on-error: true + run: | + tar -xzf compile-artifact.tar.gz + rm compile-artifact.tar.gz - name: Run tests env: ${{ fromJSON(inputs.envs) }} run: | @@ -872,6 +887,10 @@ jobs: # R issues at docker environment export TZ=UTC export _R_CHECK_SYSTEM_CLOCK_=FALSE + if [ "${{ steps.extract-precompiled.outcome }}" = "success" ]; then + export SKIP_SCALA_BUILD=true + echo "Reusing precompiled artifact, skipping local SBT build." + fi ./dev/run-tests --parallelism 1 --modules sparkr - name: Upload test results to report if: always() From 52d3c2ec69d7ca07ab0ddddb45a601970ef9ccb2 Mon Sep 17 00:00:00 2001 From: TongWei Date: Tue, 12 May 2026 14:32:07 +0800 Subject: [PATCH 104/286] [SPARK-56793][K8S] Avoid cluster-wide LIST in executor pods polling ### What changes were proposed in this pull request? Scope the executor pod LIST issued by `ExecutorPodsPollingSnapshotSource` to the configured Kubernetes namespace by inserting `.inNamespace(namespace)` between `.pods()` and the label filters. ### Why are the changes needed? Without `.inNamespace(...)` the fabric8 client issues a cluster-wide LIST against the K8s API server. Other paths in the K8s scheduler module (e.g. `KubernetesClusterSchedulerBackend.doKillExecutors`, `ExecutorPodsLifecycleManager`) already scope their pod operations to the configured namespace; the polling source was inconsistent. A cluster-wide LIST: - fails under the typical least-privilege deployment where the driver ServiceAccount is bound to a namespaced Role rather than a ClusterRole; - causes unnecessary load and broadens the visibility surface even when ClusterRole permissions are granted. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Updated `ExecutorPodsPollingSnapshotSourceSuite` to mock the new `.inNamespace(...)` link in the chain. Ran the full `resource-managers/kubernetes/core` test module: 344 tests across 42 suites, all passing. ### Was this patch authored or co-authored using generative AI tooling? Yes, Generated-by: Claude Code 4.7 Closes #55754 from TongWei1105/spark-k8s-scope-poller-namespace. Authored-by: TongWei Signed-off-by: Cheng Pan (cherry picked from commit 73df934271edf8db5104810796594d505b353162) Signed-off-by: Cheng Pan --- .../cluster/k8s/ExecutorPodsPollingSnapshotSource.scala | 2 ++ .../k8s/ExecutorPodsPollingSnapshotSourceSuite.scala | 7 ++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSource.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSource.scala index 4ed34ec3e4c00..3d2822e5eb518 100644 --- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSource.scala +++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSource.scala @@ -47,6 +47,7 @@ class ExecutorPodsPollingSnapshotSource( private val pollingInterval = conf.get(KUBERNETES_EXECUTOR_API_POLLING_INTERVAL) private val pollingEnabled = conf.get(KUBERNETES_EXECUTOR_ENABLE_API_POLLING) + private val namespace = conf.get(KUBERNETES_NAMESPACE) private var pollingFuture: Future[_] = _ @@ -76,6 +77,7 @@ class ExecutorPodsPollingSnapshotSource( logDebug(s"Resynchronizing full executor pod state from Kubernetes.") val pods = kubernetesClient .pods() + .inNamespace(namespace) .withLabel(SPARK_APP_ID_LABEL, applicationId) .withLabel(SPARK_ROLE_LABEL, SPARK_POD_EXECUTOR_ROLE) .withoutLabel(SPARK_EXECUTOR_INACTIVE_LABEL, "true") diff --git a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSourceSuite.scala b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSourceSuite.scala index e0016a2ae0503..71c187a9caf83 100644 --- a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSourceSuite.scala +++ b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSourceSuite.scala @@ -43,6 +43,9 @@ class ExecutorPodsPollingSnapshotSourceSuite extends SparkFunSuite with BeforeAn @Mock private var podOperations: PODS = _ + @Mock + private var namespacedPodOperations: PODS_WITH_NAMESPACE = _ + @Mock private var appIdLabeledPods: LABELED_PODS = _ @@ -62,7 +65,9 @@ class ExecutorPodsPollingSnapshotSourceSuite extends SparkFunSuite with BeforeAn MockitoAnnotations.openMocks(this).close() pollingExecutor = new DeterministicScheduler() when(kubernetesClient.pods()).thenReturn(podOperations) - when(podOperations.withLabel(SPARK_APP_ID_LABEL, TEST_SPARK_APP_ID)) + when(podOperations.inNamespace(defaultConf.get(KUBERNETES_NAMESPACE))) + .thenReturn(namespacedPodOperations) + when(namespacedPodOperations.withLabel(SPARK_APP_ID_LABEL, TEST_SPARK_APP_ID)) .thenReturn(appIdLabeledPods) when(appIdLabeledPods.withLabel(SPARK_ROLE_LABEL, SPARK_POD_EXECUTOR_ROLE)) .thenReturn(executorRoleLabeledPods) From 85b593f21b7500135555aceb3679fa645b8f1e82 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Tue, 12 May 2026 09:22:12 -0700 Subject: [PATCH 105/286] [SPARK-56833][TESTS] Add `-XX:+EnableDynamicAgentLoading` to test JVM options to suppress JEP 451 warnings ### What changes were proposed in this pull request? Add `-XX:+EnableDynamicAgentLoading` to the test-only JVM options in both SBT (`project/SparkBuild.scala`) and Maven (`pom.xml`). ### Why are the changes needed? Per [JEP 451: Prepare to Disallow the Dynamic Loading of Agents](https://openjdk.org/jeps/451), JDK 21+ warns when a Java agent is loaded dynamically (e.g. Mockito self-attaching ByteBuddy), and a future release will disallow it by default. Setting this flag on the test JVM silences the warning today and keeps tests working once the default flips. Production JVM options are intentionally untouched since Spark itself does not dynamically attach agents. Currently, Apache Spark CIs show the following message at ERROR level. - Java 21: https://github.com/apache/spark/actions/runs/25592460535/job/75172442863 - Java 25: https://github.com/apache/spark/actions/runs/25664584995/job/75333859273 ``` [info] Test run started (JUnit Jupiter) [info] Test org.apache.spark.network.TransportRequestHandlerSuite#handleMergedBlockMetaRequest() started [error] Mockito is currently self-attaching to enable the inline-mock-maker. This will no longer work in future releases of the JDK. Please add Mockito as an agent to your build as described in Mockito's documentation: https://javadoc.io/doc/org.mockito/mockito-core/latest/org.mockito/org/mockito/Mockito.html#0.3 WARNING: A Java agent has been loaded dynamically (/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/bytebuddy/byte-buddy-agent/1.18.4/byte-buddy-agent-1.18.4.jar) WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information WARNING: Dynamic loading of agents will be disallowed by default in a future release OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BEFORE** ``` $ build/sbt "network-common/testOnly *.TransportRequestHandlerSuite" > /dev/null Using SPARK_LOCAL_IP=localhost WARNING: A terminally deprecated method in sun.misc.Unsafe has been called WARNING: sun.misc.Unsafe::objectFieldOffset has been called by scala.runtime.LazyVals$ (file:/Users/dongjoon/APACHE/spark-merge/build/sbt-launch-1.12.8.jar) WARNING: Please consider reporting this to the maintainers of class scala.runtime.LazyVals$ WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release WARNING: Using incubator modules: jdk.incubator.vector WARNING: package sun.security.action not in java.base WARNING: A Java agent has been loaded dynamically (/Users/dongjoon/.m2/repository/net/bytebuddy/byte-buddy-agent/1.18.4/byte-buddy-agent-1.18.4.jar) WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information WARNING: Dynamic loading of agents will be disallowed by default in a future release OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended WARNING: A terminally deprecated method in sun.misc.Unsafe has been called WARNING: sun.misc.Unsafe::allocateMemory has been called by io.netty.util.internal.PlatformDependent0$2 (file:/Users/dongjoon/.m2/repository/io/netty/netty-common/4.2.12.Final/netty-common-4.2.12.Final.jar) WARNING: Please consider reporting this to the maintainers of class io.netty.util.internal.PlatformDependent0$2 WARNING: sun.misc.Unsafe::allocateMemory will be removed in a future release ``` **AFTER** ``` $ build/sbt "network-common/testOnly *.TransportRequestHandlerSuite" > /dev/null Using SPARK_LOCAL_IP=localhost WARNING: A terminally deprecated method in sun.misc.Unsafe has been called WARNING: sun.misc.Unsafe::objectFieldOffset has been called by scala.runtime.LazyVals$ (file:/Users/dongjoon/APACHE/spark-merge/.claude/worktrees/youthful-sanderson-0fc91a/build/sbt-launch-1.12.8.jar) WARNING: Please consider reporting this to the maintainers of class scala.runtime.LazyVals$ WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release WARNING: Using incubator modules: jdk.incubator.vector WARNING: package sun.security.action not in java.base OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended WARNING: A terminally deprecated method in sun.misc.Unsafe has been called WARNING: sun.misc.Unsafe::allocateMemory has been called by io.netty.util.internal.PlatformDependent0$2 (file:/Users/dongjoon/.m2/repository/io/netty/netty-common/4.2.12.Final/netty-common-4.2.12.Final.jar) WARNING: Please consider reporting this to the maintainers of class io.netty.util.internal.PlatformDependent0$2 WARNING: sun.misc.Unsafe::allocateMemory will be removed in a future release ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 (1M context) Closes #55825 from dongjoon-hyun/SPARK-56833. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun (cherry picked from commit 08a29819bf4093fb756f3160b49e5f99500323a5) Signed-off-by: Dongjoon Hyun --- pom.xml | 1 + project/SparkBuild.scala | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 8e85171068048..57faf6d781997 100644 --- a/pom.xml +++ b/pom.xml @@ -342,6 +342,7 @@ -Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE -Dio.netty.noUnsafe=false --enable-native-access=ALL-UNNAMED + -XX:+EnableDynamicAgentLoading 3.5.7 9.6.0 diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index 66947a58ac590..49556aa1df182 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -1950,7 +1950,8 @@ object TestSettings { "-Dio.netty.allocator.type=pooled", "-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE", "-Dio.netty.noUnsafe=false", - "--enable-native-access=ALL-UNNAMED").mkString(" ") + "--enable-native-access=ALL-UNNAMED", + "-XX:+EnableDynamicAgentLoading").mkString(" ") s"-Xmx$heapSize -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:ReservedCodeCacheSize=128m -Dfile.encoding=UTF-8 $extraTestJavaArgs" .split(" ").toSeq }, From defde6b580d493aa87fce8b33cadd3f5054444a9 Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Wed, 13 May 2026 00:38:43 +0800 Subject: [PATCH 106/286] [SPARK-56799][SQL] Search and highlight nodes in SQL plan visualization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? This PR adds an in-graph node search to the SQL execution detail page, sitting next to the zoom toolbar introduced in SPARK-56792. Behavior: - A magnifying-glass button in the plan-viz toolbar (or the `/` keyboard shortcut while hovering the plan) opens a compact search input with a match counter and prev/next/close buttons. - Typing performs a case-insensitive substring match against the operator name (`SparkPlanGraphNode.name` for plain nodes, `SparkPlanGraphCluster.name` for WholeStageCodegen clusters). - Matches are highlighted with an orange outline; the active match is filled with the existing 'linked' accent color and the viewport is zoomed to fit it. `Enter` / `Shift+Enter` and the up/down buttons cycle through matches in DOM order; the viewport pans smoothly. - Non-matching nodes and clusters are dimmed to opacity 0.3, but a cluster that contains a match is never dimmed (so the matched child stays visible). When the query has no matches, the plan is left fully visible and the counter shows '0/0' in red, mirroring familiar find-in-page UX. - `Esc` (or the close button) clears the search and collapses the toolbar without resetting the user's manual zoom/pan position. - Toggling detailed mode (which re-parses the dot file and rebuilds the SVG) automatically reapplies the active query against the new DOM. Implementation notes: - Matching uses `getNodeDetails()[domId].name` so detailed-mode HTML labels (which embed metric tables) are not searched as raw HTML. - Cluster ancestors of a match are tracked via the graphlib instance's `parent(v)` to drive the 'do not dim' rule. - An `!important` on the dimming rule is required because dagre-d3 writes an inline `opacity: 1` on each `` / `` at render time. ### Why are the changes needed? For wide SparkPlans (large joins, AQE plans with many shuffle/exchange nodes, generated WholeStageCodegen clusters) it is currently hard to locate a specific operator. With zoom/pan from SPARK-56792 the plan can be navigated, but the user still needs a way to ask 'where is the BroadcastHashJoin in this graph?'. ### Does this PR introduce _any_ user-facing change? Yes. A new search control appears in the SQL execution detail page's plan-viz toolbar (next to the existing zoom controls). No public APIs, configs, metrics, or persisted data are affected. Screenshots are attached in a follow-up comment. ### How was this patch tested? - `build/sbt sql/Compile/compile sql/scalastyle` - `dev/lint-js` - `node --check` on the modified JS - Manual smoke test with a multi-stage join + aggregate query, verified in both basic and detailed modes: - toolbar collapses/expands as expected - typing 'hash' matches BroadcastHashJoin + HashAggregate (×2), counter shows 1/3, viewport zooms to first match - cycling with Enter/up/down updates the active match and pans - non-matching nodes (Project, Exchange, AQEShuffleRead, LocalTableScan, BroadcastExchange) dim to 0.3 - the WholeStageCodegen cluster containing matched HashAggregate is not dimmed - toggling detailed mode preserves the search - typing a non-matching string shows '0/0' in red and leaves the plan fully visible - Esc clears highlights and collapses the toolbar without resetting the zoom level ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot CLI 1.0.44 with Claude Opus 4.7 Closes #55778 from yaooqinn/SPARK-56799. Authored-by: Kent Yao Signed-off-by: Kent Yao (cherry picked from commit bae8a599dead4299c334797e68c48f2f87c210fe) Signed-off-by: Kent Yao --- .../sql/execution/ui/static/spark-sql-viz.css | 43 +++ .../sql/execution/ui/static/spark-sql-viz.js | 263 ++++++++++++++++++ .../sql/execution/ui/ExecutionPage.scala | 21 ++ 3 files changed, 327 insertions(+) diff --git a/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.css b/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.css index dd0c9fc81bd57..262b2f7332426 100644 --- a/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.css +++ b/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.css @@ -23,6 +23,7 @@ --spark-sql-selected-fill: #E25A1CFF; --spark-sql-selected-stroke: #317EACFF; --spark-sql-linked-fill: #FFC106FF; + --spark-sql-search-stroke: #FD7E14; } [data-bs-theme="dark"] { --spark-sql-cluster-fill: #1a5276; @@ -32,6 +33,7 @@ --spark-sql-selected-fill: #c0470fff; --spark-sql-selected-stroke: #5dade2ff; --spark-sql-linked-fill: #d4a00aff; + --spark-sql-search-stroke: #ffa94d; } svg g.label { @@ -166,6 +168,8 @@ svg path.linked { top: 8px; right: 16px; z-index: 10; + display: flex; + align-items: center; } .plan-viz-zoom-toolbar #plan-viz-zoom-level { @@ -173,3 +177,42 @@ svg path.linked { min-width: 3.25rem; font-variant-numeric: tabular-nums; } + +#plan-viz-search-expanded { + width: auto; +} + +#plan-viz-search-input { + width: 12rem; +} + +#plan-viz-search-count { + min-width: 4.5rem; + justify-content: center; + font-variant-numeric: tabular-nums; + font-size: 0.75rem; +} + +#plan-viz-search-count.no-match { + color: var(--bs-danger); +} + +/* Search result highlight: outline matched node/cluster with the search color. */ +svg g.node rect.search-match, +svg g.cluster.search-match > rect { + stroke: var(--spark-sql-search-stroke); + stroke-width: 3px; +} + +/* Currently active match: keep the outline visible and add an accent fill. */ +svg g.node rect.search-match.search-active { + fill: var(--spark-sql-linked-fill); +} + +/* Dim non-matching nodes/clusters when a search is active. + `!important` is needed because dagre-d3 writes an inline `opacity: 1` + style on each / at render time. */ +svg g.node.search-dimmed, +svg g.cluster.search-dimmed { + opacity: 0.3 !important; +} diff --git a/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.js b/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.js index 037877f0dda6e..81d4562cbb35d 100644 --- a/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.js +++ b/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.js @@ -32,6 +32,12 @@ var cachedNodeDetails = null; // d3.zoom behavior for the current SVG; reinitialized on each (re)render. var planVizZoom = null; +// Current dagre graph; kept so node search can re-iterate names without re-parsing dot. +var currentPlanGraph = null; + +// Node search state. matches[] is in DOM (top-down) order, index is the active match. +var planVizSearchState = { query: "", matches: [], index: -1 }; + function shouldRenderPlanViz() { return planVizContainer().selectAll("svg").empty(); } @@ -47,6 +53,7 @@ function renderPlanViz() { var graph = zoomLayer.append("g"); var g = graphlibDot.read(dot); + currentPlanGraph = g; preprocessGraphLayout(g); var renderer = new dagreD3.render(); renderer(graph, g); @@ -64,6 +71,7 @@ function renderPlanViz() { postprocessForAdditionalMetrics(); setupDetailedLabelsToggle(); setupZoomAndPan(svg, zoomLayer); + reapplyPlanVizSearch(); } /* -------------------- * @@ -679,6 +687,7 @@ function rerenderWithDetailedLabels() { var graph = zoomLayer.append("g"); var g = graphlibDot.read(dot); + currentPlanGraph = g; // If detailed mode, inject HTML labels with metrics tables var detailed = document.getElementById("detailed-labels-checkbox"); @@ -714,6 +723,7 @@ function rerenderWithDetailedLabels() { resizeSvg(svg); postprocessForAdditionalMetrics(); setupZoomAndPan(svg, zoomLayer); + reapplyPlanVizSearch(); } /* ---------------------- * @@ -776,6 +786,257 @@ function planVizZoomReset() { } } +/* ---------------------- * + * | Node search | * + * ---------------------- */ + +/* + * Wire the search toolbar (toggle, input, navigation, close) and the global + * `/` shortcut. Idempotent: subsequent calls do nothing. + */ +function setupPlanVizSearch() { + var toggle = document.getElementById("plan-viz-search-toggle"); + var input = document.getElementById("plan-viz-search-input"); + var prevBtn = document.getElementById("plan-viz-search-prev"); + var nextBtn = document.getElementById("plan-viz-search-next"); + var closeBtn = document.getElementById("plan-viz-search-close"); + if (!toggle || !input || !prevBtn || !nextBtn || !closeBtn) return; + if (toggle.dataset.searchWired === "true") return; + toggle.dataset.searchWired = "true"; + + toggle.addEventListener("click", function () { + expandPlanVizSearch(); + }); + closeBtn.addEventListener("click", function () { + collapsePlanVizSearch(); + }); + + var debounceTimer = null; + input.addEventListener("input", function () { + if (debounceTimer) clearTimeout(debounceTimer); + debounceTimer = setTimeout(function () { + runPlanVizSearch(input.value, true); + }, 80); + }); + + input.addEventListener("keydown", function (event) { + if (event.key === "Enter") { + event.preventDefault(); + planVizSearchGoTo(event.shiftKey ? -1 : 1); + } else if (event.key === "Escape") { + event.preventDefault(); + collapsePlanVizSearch(); + } + }); + + prevBtn.addEventListener("click", function () { planVizSearchGoTo(-1); }); + nextBtn.addEventListener("click", function () { planVizSearchGoTo(1); }); + + document.addEventListener("keydown", function (event) { + if (event.ctrlKey || event.metaKey || event.altKey) return; + if (event.key !== "/") return; + var tag = event.target && event.target.tagName; + if (tag === "INPUT" || tag === "TEXTAREA" || event.target.isContentEditable) { + return; + } + var graphEl = document.getElementById("plan-viz-graph"); + if (!graphEl || !graphEl.matches(":hover")) return; + event.preventDefault(); + expandPlanVizSearch(); + }); +} + +function expandPlanVizSearch() { + var collapsed = document.getElementById("plan-viz-search-collapsed"); + var expanded = document.getElementById("plan-viz-search-expanded"); + var input = document.getElementById("plan-viz-search-input"); + if (!collapsed || !expanded || !input) return; + collapsed.classList.add("d-none"); + expanded.classList.remove("d-none"); + input.value = planVizSearchState.query; + input.focus(); + input.select(); +} + +function collapsePlanVizSearch() { + var collapsed = document.getElementById("plan-viz-search-collapsed"); + var expanded = document.getElementById("plan-viz-search-expanded"); + var input = document.getElementById("plan-viz-search-input"); + clearPlanVizSearchHighlights(); + planVizSearchState.query = ""; + planVizSearchState.matches = []; + planVizSearchState.index = -1; + updatePlanVizSearchCount(); + if (input) input.value = ""; + if (collapsed) collapsed.classList.remove("d-none"); + if (expanded) expanded.classList.add("d-none"); +} + +/* + * Recompute matches against the current query, update DOM classes, and zoom + * to the first match. When `zoomToFirst` is false (e.g. re-applying after a + * detailed-mode rerender), the viewport is left untouched. + */ +function runPlanVizSearch(rawQuery, zoomToFirst) { + var query = (rawQuery || "").trim(); + planVizSearchState.query = query; + planVizSearchState.matches = []; + planVizSearchState.index = -1; + + clearPlanVizSearchHighlights(); + + if (query === "" || !currentPlanGraph) { + updatePlanVizSearchCount(); + return; + } + + var lower = query.toLowerCase(); + var nodeDetails = getNodeDetails(); + var matchedDomIds = Object.create(null); + var ancestorsOfMatch = Object.create(null); + + currentPlanGraph.nodes().forEach(function (v) { + var node = currentPlanGraph.node(v); + if (!node) return; + var domId = node.id || (node.isCluster ? v : "node" + v); + var displayName; + if (nodeDetails[domId] && nodeDetails[domId].name) { + displayName = String(nodeDetails[domId].name); + } else { + displayName = String(node.label || ""); + } + if (displayName.toLowerCase().indexOf(lower) >= 0) { + matchedDomIds[domId] = true; + // Walk up the compound-graph hierarchy so we don't dim a cluster that + // contains a match (which would visually hide the matched child). + var parent = currentPlanGraph.parent(v); + while (parent) { + var parentNode = currentPlanGraph.node(parent); + var parentDomId = (parentNode && parentNode.id) || parent; + ancestorsOfMatch[parentDomId] = true; + parent = currentPlanGraph.parent(parent); + } + } + }); + + var svg = planVizContainer().select("svg"); + var nodeEls = svg.selectAll("g.node, g.cluster").nodes(); + var orderedMatches = []; + var anyMatch = false; + Object.keys(matchedDomIds).forEach(function () { anyMatch = true; }); + nodeEls.forEach(function (el) { + if (matchedDomIds[el.id]) { + orderedMatches.push(el.id); + var rect = el.querySelector(":scope > rect"); + if (rect) rect.classList.add("search-match"); + if (el.classList.contains("cluster")) el.classList.add("search-match"); + } else if (anyMatch && !ancestorsOfMatch[el.id]) { + // Only dim when at least one match exists; otherwise leave the plan + // fully visible so the user can adjust their query without obscuring + // context (matches familiar find-in-page UX). + el.classList.add("search-dimmed"); + } + }); + + planVizSearchState.matches = orderedMatches; + if (orderedMatches.length > 0) { + planVizSearchState.index = 0; + markActiveMatch(orderedMatches[0]); + if (zoomToFirst) zoomToNode(orderedMatches[0]); + } + updatePlanVizSearchCount(); +} + +function planVizSearchGoTo(delta) { + var matches = planVizSearchState.matches; + if (matches.length === 0) return; + var idx = (planVizSearchState.index + delta + matches.length) % matches.length; + planVizSearchState.index = idx; + markActiveMatch(matches[idx]); + zoomToNode(matches[idx]); + updatePlanVizSearchCount(); +} + +function markActiveMatch(domId) { + planVizContainer().selectAll("rect.search-active") + .classed("search-active", false); + if (!domId) return; + var el = document.getElementById(domId); + if (!el) return; + var rect = el.querySelector(":scope > rect"); + if (rect) rect.classList.add("search-active"); +} + +function clearPlanVizSearchHighlights() { + var container = planVizContainer(); + container.selectAll(".search-match").classed("search-match", false); + container.selectAll(".search-active").classed("search-active", false); + container.selectAll(".search-dimmed").classed("search-dimmed", false); +} + +function updatePlanVizSearchCount() { + var el = document.getElementById("plan-viz-search-count"); + if (!el) return; + var matches = planVizSearchState.matches; + if (planVizSearchState.query === "") { + el.textContent = ""; + el.classList.remove("no-match"); + } else if (matches.length === 0) { + el.textContent = "0/0"; + el.classList.add("no-match"); + } else { + el.textContent = (planVizSearchState.index + 1) + "/" + matches.length; + el.classList.remove("no-match"); + } +} + +/* Center and scale the viewport on the given DOM element using d3-zoom. */ +function zoomToNode(domId) { + var el = document.getElementById(domId); + if (!el || !planVizZoom) return; + var svg = planVizContainer().select("svg"); + var svgNode = svg.node(); + if (!svgNode) return; + var vb = svgNode.viewBox && svgNode.viewBox.baseVal; + if (!vb || vb.width === 0 || vb.height === 0) return; + + var bbox; + try { + bbox = el.getBBox(); + } catch (e) { + return; + } + if (!bbox || bbox.width === 0 || bbox.height === 0) return; + + // Aim to fill ~50% of the viewport so the matched node is prominent but + // still in context. Clamp to the configured zoom range. + var scale = Math.min( + vb.width / bbox.width / 2, + vb.height / bbox.height / 2, + PlanVizConstants.zoomMax + ); + scale = Math.max(scale, PlanVizConstants.zoomMin); + + var bcx = bbox.x + bbox.width / 2; + var bcy = bbox.y + bbox.height / 2; + var vcx = vb.x + vb.width / 2; + var vcy = vb.y + vb.height / 2; + var transform = d3.zoomIdentity + .translate(vcx - bcx * scale, vcy - bcy * scale) + .scale(scale); + + svg.transition().duration(400).call(planVizZoom.transform, transform); +} + +/* + * After a render or detailed-mode toggle, reapply the active query against the + * fresh DOM so highlights survive re-layout. No-ops when no search is active. + */ +function reapplyPlanVizSearch() { + if (!planVizSearchState.query) return; + runPlanVizSearch(planVizSearchState.query, false); +} + document.addEventListener("DOMContentLoaded", function () { if (shouldRenderPlanViz()) { renderPlanViz(); @@ -798,6 +1059,8 @@ document.addEventListener("DOMContentLoaded", function () { zoomResetBtn.addEventListener("click", planVizZoomReset); } + setupPlanVizSearch(); + // Keyboard shortcuts when the SVG is focused or the user hovers over it. document.addEventListener("keydown", function (event) { if (event.ctrlKey || event.metaKey || event.altKey) return; diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala index 827e0f92dfc91..9ff410e829e27 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala @@ -169,6 +169,27 @@ class ExecutionPage(parent: SQLTab) extends WebUIPage("execution") with Logging Show metrics in graph nodes (detailed mode)

+
+ +
+
+ + + + + +
From ff7b7b60f23809ddf381ce896896a0f52842392b Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Tue, 12 May 2026 09:54:19 -0700 Subject: [PATCH 107/286] [SPARK-56815][DOCS][FOLLOWUP] Add Java 25 to `building-spark.md` ### What changes were proposed in this pull request? Update `docs/building-spark.md` from `Java 17/21` to `Java 17/21/25`. This is a follow-up of - apache/spark#55798 ### Why are the changes needed? `docs/index.md` already states "Spark runs on Java 17/21/25", but `docs/building-spark.md` was missed and still says `Java 17/21`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? N/A. Documentation-only change. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes #55827 from dongjoon-hyun/SPARK-56815. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun (cherry picked from commit 2603abfa3c4437337f17fd33bda4111d4ae09c9f) Signed-off-by: Dongjoon Hyun --- docs/building-spark.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-spark.md b/docs/building-spark.md index e9eb0b22271aa..0cae594af2e3e 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -27,7 +27,7 @@ license: | ## Apache Maven The Maven-based build is the build of reference for Apache Spark. -Building Spark using Maven requires Maven 3.9.15 and Java 17/21. +Building Spark using Maven requires Maven 3.9.15 and Java 17/21/25. Spark requires Scala 2.13; support for Scala 2.12 was removed in Spark 4.0.0. ### Setting up Maven's Memory Usage From 52f267395bf2e71c984f4c565820637e0e9d16ad Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Tue, 12 May 2026 11:21:43 -0700 Subject: [PATCH 108/286] [SPARK-56834][K8S] Use Java `25-jre` instead of `21-jre` image in K8s Dockerfile ### What changes were proposed in this pull request? This PR aims to use Java 25 image in the Spark K8s Dockerfile. ### Why are the changes needed? As a part of [SPARK-51167 Build and Run Spark on Java 25](https://issues.apache.org/jira/browse/SPARK-51167), this PR uses the latest LTS Java release as the default JVM docker base image in Apache Spark 4.2.0. - apache/spark#55798 - apache/spark#55827 ### Does this PR introduce _any_ user-facing change? - Yes because the default Java version is changed. - However, a user still change this back via `-b java_image_tag=17` and the Apache Spark 4.2.0 provides the same capability for Java 17/21/25. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes #55829 from dongjoon-hyun/SPARK-56834. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun (cherry picked from commit 7a01891b31ea95ca4d4266947099d11cd7f5ddf5) Signed-off-by: Dongjoon Hyun --- .../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile index 84cba5568d27c..4af2359237d36 100644 --- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile +++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile @@ -15,7 +15,7 @@ # limitations under the License. # ARG java_image_name=azul/zulu-openjdk -ARG java_image_tag=21-jre +ARG java_image_tag=25-jre FROM ${java_image_name}:${java_image_tag} LABEL org.opencontainers.image.authors="Apache Spark project " From 716f3d2e0b48af1b254dde669fc35cdbe474edfe Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Tue, 12 May 2026 12:21:45 -0700 Subject: [PATCH 109/286] [SPARK-56835][K8S][DOCS][INFRA] Upgrade Volcano to 1.14.2 ### What changes were proposed in this pull request? This PR aims to upgrade `Volcano` to 1.14.2 in K8s integration test document and GA job. ### Why are the changes needed? To use the latest version for testing and documentation for Apache Spark 4.2.0. - https://github.com/volcano-sh/volcano/releases/tag/v1.14.2 - [CVE-2026-44247 Volcano's webhook server vulnerable to OOM due to unbounded HTTP request body size](https://github.com/advisories/GHSA-8wxp-xxp2-rcgx) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: `Claude Opus 4.7 (1M context)` Closes #55830 from dongjoon-hyun/SPARK-56835. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun (cherry picked from commit 42422741167664dc0d2ad535279b2a4c67906539) Signed-off-by: Dongjoon Hyun --- .github/workflows/build_and_test.yml | 2 +- docs/running-on-kubernetes.md | 4 ++-- resource-managers/kubernetes/integration-tests/README.md | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 3f3400153cde7..2c7a71fc42582 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -1595,7 +1595,7 @@ jobs: elif [[ "${{ inputs.branch }}" == 'branch-4.0' ]]; then kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.11.0/installer/volcano-development.yaml || true else - kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.1/installer/volcano-development.yaml || true + kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml || true fi eval $(minikube docker-env) build/sbt -Phadoop-3 -Psparkr -Pkubernetes -Pvolcano -Pkubernetes-integration-tests -Dspark.kubernetes.test.volcanoMaxConcurrencyJobNum=1 -Dtest.exclude.tags=local "kubernetes-integration-tests/test" diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index 558eb75ef3d3a..aa753e259bcc7 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -2053,10 +2053,10 @@ Spark allows users to specify a custom Kubernetes schedulers. #### Using Volcano as Customized Scheduler for Spark on Kubernetes ##### Prerequisites -* Spark on Kubernetes with [Volcano](https://volcano.sh/en) as a custom scheduler is supported since Spark v3.3.0 and Volcano v1.7.0. Below is an example to install Volcano 1.14.1: +* Spark on Kubernetes with [Volcano](https://volcano.sh/en) as a custom scheduler is supported since Spark v3.3.0 and Volcano v1.7.0. Below is an example to install Volcano 1.14.2: ```bash - kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.1/installer/volcano-development.yaml + kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml ``` ##### Build diff --git a/resource-managers/kubernetes/integration-tests/README.md b/resource-managers/kubernetes/integration-tests/README.md index 2b54f8eabd09e..0d6279ccdba44 100644 --- a/resource-managers/kubernetes/integration-tests/README.md +++ b/resource-managers/kubernetes/integration-tests/README.md @@ -336,11 +336,11 @@ You can also specify your specific dockerfile to build JVM/Python/R based image ## Requirements - A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases. -- Volcano v1.14.1. +- Volcano v1.14.2. ## Installation - kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.1/installer/volcano-development.yaml + kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml ## Run tests @@ -361,5 +361,5 @@ You can also specify `volcano` tag to only run Volcano test: ## Cleanup Volcano - kubectl delete -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.1/installer/volcano-development.yaml + kubectl delete -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml From f619d4d1922ac262adebaaa6cb575233db6a62ac Mon Sep 17 00:00:00 2001 From: Uros Bojanic Date: Wed, 13 May 2026 08:58:33 +0800 Subject: [PATCH 110/286] [SPARK-56813][DOCS] Refine the documentation for geospatial types and functions ### What changes were proposed in this pull request? Tighten the geospatial documentation: - `sql-ref-geospatial-types.md`: update the documentation for supported ST functions. - `sql-ref-datatypes.md`: drop unparameterized type syntax from the SQL and PySpark. - `sql-ref-functions-builtin.md`: surface `st_funcs` group as **Geospatial ST Functions**. - `sql-migration-guide.md`: note that geospatial types and ST functions are enabled since 4.2. - etc. ### Why are the changes needed? Fix gaps and accuracy for geospatial documentation. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests suffice for docs only changes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7. Closes #55790 from uros-db/geo-docs-refine. Authored-by: Uros Bojanic Signed-off-by: Wenchen Fan (cherry picked from commit e648859ac6c3a9918c81cc95b61b658c6b3dca54) Signed-off-by: Wenchen Fan --- docs/sql-ref-datatypes.md | 16 ++++++++-------- docs/sql-ref-functions-builtin.md | 5 +++++ docs/sql-ref-geospatial-types.md | 30 ++++++++++++++++++++---------- sql/gen-sql-functions-docs.py | 3 ++- 4 files changed, 35 insertions(+), 19 deletions(-) diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md index 743ad4e3abb22..0ae05d8f46bef 100644 --- a/docs/sql-ref-datatypes.md +++ b/docs/sql-ref-datatypes.md @@ -95,8 +95,8 @@ Spark SQL and DataFrames support the following data types: * Spatial types Spatial objects as defined in the [OGC Simple Feature Access](https://portal.ogc.org/files/?artifact_id=25355) specification. - - `GeometryType`: Represents GEOMETRY values—spatial objects in a Cartesian coordinate system. The type can be fixed to a single SRID, e.g. `geometry(4326)`, or allow mixed SRIDs with `geometry(any)`. Default SRID when not specified is 4326 (WGS 84). - - `GeographyType`: Represents GEOGRAPHY values—spatial objects in a geographic coordinate system (latitude/longitude). Edge interpolation is always SPHERICAL. The type can be fixed to a single SRID, e.g. `geography(4326)`, or allow mixed SRIDs with `geography(any)`. Default SRID is 4326 (WGS 84). + - `GeometryType`: Represents GEOMETRY values, spatial objects in a Cartesian coordinate system. The type can be fixed to a single SRID, e.g. `geometry(4326)`, or allow mixed SRIDs with `geometry(any)`. In SQL, `GEOMETRY` columns must always be declared with an explicit SRID or `ANY`. + - `GeographyType`: Represents GEOGRAPHY values, spatial objects in a geographic coordinate system (latitude/longitude). Edge interpolation is always SPHERICAL. The type can be fixed to a single geographic SRID, e.g. `geography(4326)`, or allow mixed SRIDs with `geography(any)`. In SQL, `GEOGRAPHY` columns must always be declared with an explicit SRID or `ANY`. For more details and built-in functions, see [Geospatial (Geometry/Geography) types](sql-ref-geospatial-types.html). * Complex types @@ -143,8 +143,8 @@ from pyspark.sql.types import * |**TimestampNTZType**|datetime.datetime|TimestampNTZType()| |**DateType**|datetime.date|DateType()| |**DayTimeIntervalType**|datetime.timedelta|DayTimeIntervalType()| -|**GeometryType**|Geometry|GeometryType() or GeometryType(*srid*)| -|**GeographyType**|Geography|GeographyType() or GeographyType(*srid*)| +|**GeometryType**|Geometry|GeometryType(*srid*)
**Note:** *srid* is required and may be an `int` or the string `"ANY"`.| +|**GeographyType**|Geography|GeographyType(*srid*)
**Note:** *srid* is required and may be an `int` or the string `"ANY"`.| |**ArrayType**|list, tuple, or array|ArrayType(*elementType*, [*containsNull*])
**Note:**The default value of *containsNull* is True.| |**MapType**|dict|MapType(*keyType*, *valueType*, [*valueContainsNull]*)
**Note:**The default value of *valueContainsNull* is True.| |**StructType**|list or tuple|StructType(*fields*)
**Note:** *fields* is a Seq of StructFields. Also, two fields with the same name are not allowed.| @@ -179,8 +179,8 @@ You can access them by doing |**TimeType**|java.time.LocalTime|TimeType| |**YearMonthIntervalType**|java.time.Period|YearMonthIntervalType| |**DayTimeIntervalType**|java.time.Duration|DayTimeIntervalType| -|**GeometryType**|org.apache.spark.sql.types.Geometry|GeometryType or GeometryType(*srid*)| -|**GeographyType**|org.apache.spark.sql.types.Geography|GeographyType or GeographyType(*srid*)| +|**GeometryType**|org.apache.spark.sql.types.Geometry|GeometryType(*srid*)| +|**GeographyType**|org.apache.spark.sql.types.Geography|GeographyType(*srid*)| |**ArrayType**|scala.collection.Seq|ArrayType(*elementType*, [*containsNull]*)
**Note:** The default value of *containsNull* is true.| |**MapType**|scala.collection.Map|MapType(*keyType*, *valueType*, [*valueContainsNull]*)
**Note:** The default value of *valueContainsNull* is true.| |**StructType**|org.apache.spark.sql.Row|StructType(*fields*)
**Note:** *fields* is a Seq of StructFields. Also, two fields with the same name are not allowed.| @@ -272,8 +272,8 @@ The following table shows the type names as well as aliases used in Spark SQL pa |**DecimalType**|DECIMAL, DEC, NUMERIC| |**YearMonthIntervalType**|INTERVAL YEAR, INTERVAL YEAR TO MONTH, INTERVAL MONTH| |**DayTimeIntervalType**|INTERVAL DAY, INTERVAL DAY TO HOUR, INTERVAL DAY TO MINUTE, INTERVAL DAY TO SECOND, INTERVAL HOUR, INTERVAL HOUR TO MINUTE, INTERVAL HOUR TO SECOND, INTERVAL MINUTE, INTERVAL MINUTE TO SECOND, INTERVAL SECOND| -|**GeometryType**|GEOMETRY or GEOMETRY(*srid*) or GEOMETRY(ANY)| -|**GeographyType**|GEOGRAPHY or GEOGRAPHY(*srid*) or GEOGRAPHY(ANY)| +|**GeometryType**|GEOMETRY(*srid*) or GEOMETRY(ANY)| +|**GeographyType**|GEOGRAPHY(*srid*) or GEOGRAPHY(ANY)| |**ArrayType**|ARRAY\| |**StructType**|STRUCT
**Note:** ':' is optional.| |**MapType**|MAP| diff --git a/docs/sql-ref-functions-builtin.md b/docs/sql-ref-functions-builtin.md index b6572609a34b8..1912a1e577d59 100644 --- a/docs/sql-ref-functions-builtin.md +++ b/docs/sql-ref-functions-builtin.md @@ -126,3 +126,8 @@ license: | {% include_api_gen generated-variant-funcs-table.html %} #### Examples {% include_api_gen generated-variant-funcs-examples.html %} + +### Geospatial ST Functions +{% include_api_gen generated-st-funcs-table.html %} +#### Examples +{% include_api_gen generated-st-funcs-examples.html %} diff --git a/docs/sql-ref-geospatial-types.md b/docs/sql-ref-geospatial-types.md index ed8b6597ae1f0..d5a9d0fece84b 100644 --- a/docs/sql-ref-geospatial-types.md +++ b/docs/sql-ref-geospatial-types.md @@ -25,8 +25,13 @@ Spark SQL supports **GEOMETRY** and **GEOGRAPHY** types for spatial data, as def | Type | Coordinate system | Typical use and notes | |------|-------------------|------------------------| -| **GEOMETRY** | Cartesian (planar) | Projected or local coordinates; planar calculations. Represents points, lines, polygons in a flat coordinate system. Suitable for Web Mercator (SRID 3857), UTM, or local grids (e.g. engineering/CAD). Default SRID in Spark is 4326. | -| **GEOGRAPHY** | Geographic (latitude/longitude) | Earth-based data; distances and areas on the sphere/ellipsoid. Coordinates in longitude and latitude (degrees). Edge interpolation is always **SPHERICAL**. Default SRID is 4326 (WGS 84). | +| **GEOMETRY** | Cartesian (planar) | Projected or local coordinates; planar calculations. Represents points, lines, polygons in a flat coordinate system. Suitable for Web Mercator (SRID 3857), UTM, or local grids (e.g. engineering/CAD). Accepts any SRID in the registry, including SRID 0 (unspecified CRS). | +| **GEOGRAPHY** | Geographic (latitude/longitude) | Earth-based data; distances and areas on the sphere/ellipsoid. Coordinates in longitude and latitude (degrees). Edge interpolation is always **SPHERICAL**. Only geographic SRIDs are accepted; the most common is 4326 (WGS 84). | + +In SQL, `GEOMETRY` and `GEOGRAPHY` columns must always be declared with an explicit SRID +(or `ANY`); see [Type Syntax in SQL](#type-syntax-in-sql) below. When a value is constructed +via `ST_GeomFromWKB(wkb)` without an explicit SRID, the value's SRID is `0` (unspecified), +while `ST_GeogFromWKB(wkb)` always returns a value with SRID 4326. #### When to use GEOMETRY vs GEOGRAPHY @@ -113,16 +118,18 @@ When parsing WKB, Spark applies the following rules. Violations result in a pars ### Built-in Geospatial (ST) Functions -Spark SQL provides scalar functions for working with GEOMETRY and GEOGRAPHY values. They are grouped under **st_funcs** in the [Built-in Functions](sql-ref-functions-builtin.html) API. +Spark SQL provides scalar functions for working with GEOMETRY and GEOGRAPHY values. The full list, +with detailed argument descriptions and examples, is on the +[Built-in Functions](sql-ref-functions-builtin.html#geospatial-st-functions) page under +**Geospatial ST Functions**. The functions provided in the current release are summarized here: | Function | Description | |----------|-------------| -| `ST_AsBinary(geo)` | Returns the GEOMETRY or GEOGRAPHY value as WKB (BINARY). | -| `ST_GeomFromWKB(wkb)` | Parses WKB and returns a GEOMETRY with default SRID 0. | -| `ST_GeomFromWKB(wkb, srid)` | Parses WKB and returns a GEOMETRY with the given SRID. | +| `ST_AsBinary(geo[, endianness])` | Returns the GEOMETRY or GEOGRAPHY value as WKB (BINARY). The optional `endianness` argument is `'NDR'` for little-endian (default) or `'XDR'` for big-endian. | +| `ST_GeomFromWKB(wkb[, srid])` | Parses WKB and returns a GEOMETRY. The optional `srid` argument sets the SRID; if omitted, the SRID is `0`. | | `ST_GeogFromWKB(wkb)` | Parses WKB and returns a GEOGRAPHY with SRID 4326. | | `ST_Srid(geo)` | Returns the SRID of the GEOMETRY or GEOGRAPHY value (NULL if input is NULL). | -| `ST_SetSrid(geo, srid)` | Returns a new GEOMETRY or GEOGRAPHY with the given SRID. | +| `ST_SetSrid(geo, srid)` | Returns a new GEOMETRY or GEOGRAPHY with the given SRID. The new SRID must be valid for the value's type. | **Examples:** @@ -130,6 +137,9 @@ Spark SQL provides scalar functions for working with GEOMETRY and GEOGRAPHY valu SELECT hex(ST_AsBinary(ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040'))); -- 0101000000000000000000F03F0000000000000040 +SELECT hex(ST_AsBinary(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040'), 'XDR')); +-- 00000000013FF00000000000004000000000000000 + SELECT ST_Srid(ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040')); -- 4326 @@ -139,9 +149,9 @@ SELECT ST_Srid(ST_SetSrid(ST_GeomFromWKB(X'0101000000000000000000F03F00000000000 ### SRID and Stored Values -* **Fixed-SRID columns**: Every value in the column must have the same SRID as the column type. Inserting a value with a different SRID can raise an error (or you can use `ST_SetSrid` to set the value’s SRID to match the column). -* **Mixed-SRID columns** (`GEOMETRY(ANY)` or `GEOGRAPHY(ANY)`): Values can have different SRIDs. Only valid SRIDs are allowed. -* **Storage**: Parquet, Delta, and Iceberg store geometry/geography with a fixed SRID per column; mixed-SRID types are for in-memory/query use. When writing to these formats, a concrete (fixed) SRID is required. +* **Fixed-SRID columns**: Every value in the column must have the same SRID as the column type. Inserting a value with a different SRID raises a `GEO_ENCODER_SRID_MISMATCH_ERROR`. Use `ST_SetSrid` to change a value's SRID to match the column. +* **Mixed-SRID columns** (`GEOMETRY(ANY)` or `GEOGRAPHY(ANY)`): Values can have different SRIDs per row. Each value must still have a valid SRID for the type; an invalid SRID raises `ST_INVALID_SRID_VALUE`. +* **Storage**: Parquet, Delta, and Iceberg store geometry/geography with a fixed SRID per column. They do not support persisting `GEOMETRY(ANY)` or `GEOGRAPHY(ANY)`; mixed-SRID types exist for in-memory/query use only. ### Supported SRIDs diff --git a/sql/gen-sql-functions-docs.py b/sql/gen-sql-functions-docs.py index 13f9ae055fa73..2ae00f6db8221 100644 --- a/sql/gen-sql-functions-docs.py +++ b/sql/gen-sql-functions-docs.py @@ -36,7 +36,8 @@ "bitwise_funcs", "conversion_funcs", "csv_funcs", "xml_funcs", "lambda_funcs", "collection_funcs", "url_funcs", "hash_funcs", "struct_funcs", - "table_funcs", "variant_funcs", "protobuf_funcs", "sketch_funcs" + "table_funcs", "variant_funcs", "protobuf_funcs", "sketch_funcs", + "st_funcs" } From 0c75fa8737c0e65b3ef2413139fa63d2dba3a82f Mon Sep 17 00:00:00 2001 From: Uros Bojanic Date: Wed, 13 May 2026 09:02:06 +0800 Subject: [PATCH 111/286] [SPARK-49671][SQL] Remove the RTRIM collation config ### What changes were proposed in this pull request? Cleanup SQL configs by removing `TRIM_COLLATION_ENABLED` and its surrounding code changes, thus eliminating the `spark.sql.collation.trim.enabled` feature flag from the codebase. ### Why are the changes needed? Collations and RTRIM are already enabled by default, and these configs are no longer needed. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests suffice. Outdated tests have been removed. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53576 from uros-db/remove-rtrim-flag. Authored-by: Uros Bojanic Signed-off-by: Wenchen Fan (cherry picked from commit a7c0d213480d98a5d18fa0931d85cf286a8f31d0) Signed-off-by: Wenchen Fan --- .../expressions/collationExpressions.scala | 5 --- .../sql/catalyst/parser/AstBuilder.scala | 6 +--- .../sql/errors/QueryCompilationErrors.scala | 7 ---- .../apache/spark/sql/internal/SQLConf.scala | 12 ------- .../errors/QueryCompilationErrorsSuite.scala | 33 ------------------- 5 files changed, 1 insertion(+), 62 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala index 9d7c2236678d1..c3db6fca6a861 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala @@ -24,7 +24,6 @@ import org.apache.spark.sql.catalyst.expressions.codegen._ import org.apache.spark.sql.catalyst.trees.TreePattern.{TreePattern, UNRESOLVED_COLLATION} import org.apache.spark.sql.catalyst.util.{AttributeNameParser, CollationFactory} import org.apache.spark.sql.errors.QueryCompilationErrors -import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.internal.types.StringTypeWithCollation import org.apache.spark.sql.types._ @@ -54,10 +53,6 @@ object CollateExpressionBuilder extends ExpressionBuilder { if (evalCollation == null) { throw QueryCompilationErrors.unexpectedNullError("collation", collationExpr) } else { - if (!SQLConf.get.trimCollationEnabled && - evalCollation.toString.toUpperCase().contains("TRIM")) { - throw QueryCompilationErrors.trimCollationNotEnabledError() - } Collate(e, UnresolvedCollation( AttributeNameParser.parseAttributeName(evalCollation.toString))) } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala index 95b21eb01b4b7..460cf816e57cc 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala @@ -3549,11 +3549,7 @@ class AstBuilder extends DataTypeAstBuilder } override def visitCollateClause(ctx: CollateClauseContext): Seq[String] = withOrigin(ctx) { - val collationName = visitMultipartIdentifier(ctx.collationName) - if (!SQLConf.get.trimCollationEnabled && collationName.last.toUpperCase().contains("TRIM")) { - throw QueryCompilationErrors.trimCollationNotEnabledError() - } - collationName + visitMultipartIdentifier(ctx.collationName) } /** diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala index e6cf2ded76220..ceb384b2f533d 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala @@ -396,13 +396,6 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat ) } - def trimCollationNotEnabledError(): Throwable = { - new AnalysisException( - errorClass = "UNSUPPORTED_FEATURE.TRIM_COLLATION", - messageParameters = Map.empty - ) - } - def trailingCommaInSelectError(origin: Origin): Throwable = { new AnalysisException( errorClass = "TRAILING_COMMA_IN_SELECT", diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 0422b424d224e..9bb5b98cfabf6 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -1141,16 +1141,6 @@ object SQLConf { .booleanConf .createWithDefault(false) - val TRIM_COLLATION_ENABLED = - buildConf("spark.sql.collation.trim.enabled") - .internal() - .doc("When enabled allows the use of trim collations which trim trailing whitespaces from" + - " strings." - ) - .version("4.0.0") - .booleanConf - .createWithDefault(true) - val COLLATION_AWARE_HASHING_ENABLED = buildConf("spark.sql.legacy.collationAwareHashFunctions") .internal() @@ -7701,8 +7691,6 @@ class SQLConf extends Serializable with Logging with SqlApiConf { def schemaLevelCollationsEnabled: Boolean = getConf(SCHEMA_LEVEL_COLLATIONS_ENABLED) - def trimCollationEnabled: Boolean = getConf(TRIM_COLLATION_ENABLED) - def adaptiveExecutionEnabled: Boolean = getConf(ADAPTIVE_EXECUTION_ENABLED) def adaptiveExecutionEnabledInStatelessStreaming: Boolean = diff --git a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala index 2c10497c190e8..c626d7183513e 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala @@ -876,39 +876,6 @@ class QueryCompilationErrorsSuite "inputTypes" -> "[\"INT\", \"STRING\", \"STRING\"]")) } - test("SPARK-49666: the trim collation feature is off without collate builder call") { - withSQLConf(SQLConf.TRIM_COLLATION_ENABLED.key -> "false") { - Seq( - "CREATE TABLE t(col STRING COLLATE EN_RTRIM_CI) USING parquet", - "CREATE TABLE t(col STRING COLLATE UTF8_LCASE_RTRIM) USING parquet", - "SELECT 'aaa' COLLATE UNICODE_LTRIM_CI" - ).foreach { sqlText => - checkError( - exception = intercept[AnalysisException](sql(sqlText)), - condition = "UNSUPPORTED_FEATURE.TRIM_COLLATION" - ) - } - } - } - - test("SPARK-49666: the trim collation feature is off with collate builder call") { - withSQLConf(SQLConf.TRIM_COLLATION_ENABLED.key -> "false") { - Seq( - "SELECT collate('aaa', 'UNICODE_RTRIM')", - "SELECT collate('aaa', 'UTF8_BINARY_RTRIM')", - "SELECT collate('aaa', 'EN_AI_RTRIM')" - ).foreach { sqlText => - checkError( - exception = intercept[AnalysisException](sql(sqlText)), - condition = "UNSUPPORTED_FEATURE.TRIM_COLLATION", - parameters = Map.empty, - context = - ExpectedContext(fragment = sqlText.substring(7), start = 7, stop = sqlText.length - 1) - ) - } - } - } - test("SPARK-50779: the object level collations feature is unsupported when flag is disabled") { withSQLConf(SQLConf.OBJECT_LEVEL_COLLATIONS_ENABLED.key -> "false") { Seq( From 6aea6016741311d35dcee124ca505494e92f6be6 Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Wed, 13 May 2026 11:51:24 +0800 Subject: [PATCH 112/286] [SPARK-56546][SQL][FOLLOWUP] Address review comments in segment-tree window frame ### What changes were proposed in this pull request? Four small cleanups in the segment-tree moving-frame window code introduced by #55422: 1. `WindowEvaluatorFactoryBase.scala` -- fix terminology in the `def processor` comment. The comment says `Keep as def (by-name)`, but `def processor(index: Int)` is a parameterized method, not a by-name parameter (`=> T`). Reword to `Keep as def (lazy / per-call)`. 2. `WindowEvaluatorFactoryBase.eligibleForSegTree` -- add a defensive `case _ => false` to the `frameType match` so future additions to the sealed `FrameType` trait do not silently throw `MatchError` at runtime. 3. `WindowEvaluatorFactoryBase.estimateMaxCachedBlocks` -- add a comment justifying the `+ 2` slack in the cached-block budget (one boundary block at each end of the frame's interval), since the magic number was not previously explained. 4. `WindowSegmentTreeSuite.scala` -- fix indentation of 11 `test(` blocks that were declared at 4-space indent (inconsistent with the file's 2-space convention and the 3 other correctly-indented tests in the same file). A separate follow-up is needed to add RANGE-frame coverage to `WindowBenchmark` -- the current benchmark is RowFrame-only -- but that requires regenerating the committed JDK 17/21/25 results files and is deferred. ### Why are the changes needed? Review-comment-style follow-ups. Pure comment / defensive-default / whitespace changes -- no behavior change. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests cover all touched code paths. The test indentation fix is whitespace-only; the comment and `case _` changes have no runtime effect. ### Was this patch authored or co-authored using generative AI tooling? Yes, Claude assisted in identifying and drafting these cleanups. Closes #55815 from cloud-fan/cloud-fan/SPARK-56546-followup. Authored-by: Wenchen Fan Signed-off-by: Wenchen Fan (cherry picked from commit 407a29c6da0a9e88c2d57226024239c10f80c94e) Signed-off-by: Wenchen Fan --- .../window/WindowEvaluatorFactoryBase.scala | 15 +++++++++---- .../window/WindowSegmentTreeSuite.scala | 22 +++++++++---------- 2 files changed, 22 insertions(+), 15 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala index cebcfd05bd244..2ae10ce9d711c 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala @@ -203,10 +203,10 @@ trait WindowEvaluatorFactoryBase { case WindowExpression(ae: AggregateExpression, _) => ae.filter case _ => None }.toArray - // Keep as `def` (by-name): the FRAME_LESS_OFFSET / UNBOUNDED_OFFSET / - // UNBOUNDED_PRECEDING_OFFSET branches do not read `processor`. Eager - // `val` construction would invoke `AggregateProcessor.apply` on - // Lag / Lead / NthValue and throw + // Keep as `def` (lazy / per-call): the FRAME_LESS_OFFSET / + // UNBOUNDED_OFFSET / UNBOUNDED_PRECEDING_OFFSET branches do not read + // `processor`. Eager `val` construction would invoke + // `AggregateProcessor.apply` on Lag / Lead / NthValue and throw // `INTERNAL_ERROR: Unsupported aggregate function`. def processor = if (functions.exists(_.isInstanceOf[PythonFuncExpression])) { null @@ -367,6 +367,7 @@ trait WindowEvaluatorFactoryBase { val frameTypeOk = frameType match { case RowFrame => true case RangeFrame => orderSpec.size == 1 + case _ => false } conf.windowSegmentTreeEnabled && frameTypeOk && @@ -395,6 +396,12 @@ trait WindowEvaluatorFactoryBase { case (IntegerLiteral(lo), CurrentRow) => Some(math.abs(lo) + 1) case _ => None } + // `ceil(W / blockSize)` is the minimum number of blocks a single frame can + // straddle; `+ 2` adds one block of slack at each end to cover the case + // where the frame's [lower, upper) interval is offset within its leftmost + // block (so the cursor temporarily holds the previous block as well) and + // the symmetric case at the right edge -- without this slack the LRU + // would thrash on the boundary blocks every time the cursor advances. w.map(ww => math.ceil(ww.toDouble / blockSize).toInt + 2).orElse(Some(8)) } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/window/WindowSegmentTreeSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/window/WindowSegmentTreeSuite.scala index ac2a04744fcfd..443d160394fac 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/window/WindowSegmentTreeSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/window/WindowSegmentTreeSuite.scala @@ -87,7 +87,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { if (out.isNullAt(0)) null else out.getInt(0) } - test("build and single-point query returns identity; full scan matches naive") { + test("build and single-point query returns identity; full scan matches naive") { withTaskContext { val values = Seq(5, 2, 9, 1, 7, 3, 4, 8, 6, 0) val tree = buildTree(values, fanout = 4, blockSize = 1024) @@ -102,7 +102,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("single-block: range query matches naive baseline for random ranges") { + test("single-block: range query matches naive baseline for random ranges") { withTaskContext { val rnd = new Random(0xC0FFEE) val values = Seq.fill(100)(rnd.nextInt(1000)) @@ -119,7 +119,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("fanout boundaries: sizes {1, F, F+1, F*F} for fanout in {2,4,8,16}") { + test("fanout boundaries: sizes {1, F, F+1, F*F} for fanout in {2,4,8,16}") { withTaskContext { val rnd = new Random(42) for (fanout <- Seq(2, 4, 8, 16)) { @@ -151,7 +151,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("identity at empty range query(k, k)") { + test("identity at empty range query(k, k)") { withTaskContext { val values = (1 to 50).reverse val tree = buildTree(values, fanout = 4, blockSize = 16) @@ -163,7 +163,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("block boundary correctness: cross-block vs single-block baseline") { + test("block boundary correctness: cross-block vs single-block baseline") { withTaskContext { val rnd = new Random(123) val values = Seq.fill(100)(rnd.nextInt(10000)) @@ -182,7 +182,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("LRU stability: same queries in different orders produce same results") { + test("LRU stability: same queries in different orders produce same results") { withTaskContext { val rnd = new Random(777) val values = Seq.fill(100)(rnd.nextInt(10000)) @@ -213,7 +213,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("cross-block: range query matches naive baseline for random ranges") { + test("cross-block: range query matches naive baseline for random ranges") { withTaskContext { val rnd = new Random(0xBEEF) val values = Seq.fill(100)(rnd.nextInt(1000)) @@ -231,7 +231,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("cross-block: multi-block level-size invariant") { + test("cross-block: multi-block level-size invariant") { withTaskContext { val rnd = new Random(31337) val fanout = 4 @@ -329,7 +329,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("D10 rebuild: second build replaces state; failed build preserves prior state") { + test("D10 rebuild: second build replaces state; failed build preserves prior state") { withTaskContext { val v1 = Seq(5, 1, 9, 3, 7, 2, 8, 4, 6, 0) val v2 = Seq(100, 200, 50, 400, 25, 600, 12, 800) @@ -380,7 +380,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { } } - test("D11 error paths: invalid ctor args and invalid query ranges") { + test("D11 error paths: invalid ctor args and invalid query ranges") { withTaskContext { // Constructor validation. intercept[IllegalArgumentException] { @@ -503,7 +503,7 @@ class WindowSegmentTreeSuite extends SparkFunSuite with LocalSparkContext { math.sqrt(sq / (n - 1)) } - test("D12 block-aligned cross-block boundaries") { + test("D12 block-aligned cross-block boundaries") { withTaskContext { val rnd = new Random(12) val numRows = 50 From fecd8c3402dc358a7c6fdafd1fda6b8639cc2090 Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Wed, 13 May 2026 15:17:54 +0800 Subject: [PATCH 113/286] [SPARK-56832][INFRA] Surface fatal javadoc errors in unidoc log summary and CI annotations ### What changes were proposed in this pull request? After the noise filters from #55605, the Documentation generation CI log is around 4K lines on a failure run. The two-line per-file `error: reference not found` diagnostics are still buried in the middle of the log, and the GitHub Actions check panel for a failed doc-gen job only surfaces `Process completed with exit code 1`. Reviewers end up scrolling the raw log to find what actually broke. This PR is purely additive in `docs/_plugins/build_api_docs.rb` -- no existing log lines are dropped. After the unidoc pipe closes: 1. A trailing `Fatal javadoc errors (N):` block is printed, listing each captured diagnostic with file, line, and message. 2. One `::error file=,line=,title=javadoc::` GitHub Actions workflow command is emitted per diagnostic, so they appear as inline annotations on the PR check panel instead of as a single opaque `exit code 1`. Diagnostics are captured strictly within the Standard Doclet phase bracketed by `Building tree for all the packages and classes...` and `Building index for all classes...`, which is where doclint emits the build-failing diagnostics that count toward javadoc's exit code. Source-loading `error:` chatter outside that window is excluded -- it's already non-fatal and matches what javadoc's own `N errors` summary line counts. As a self-check, the captured count is compared against javadoc's own `N errors` summary line. If they diverge -- e.g. because a future JDK changes the Standard Doclet phase wording -- a `::warning::` workflow command is emitted so the drift is surfaced without silently masking real failures. ### Why are the changes needed? PR #55605 made the doc-gen log small enough to read, but the failure path is still discoverable only via grep. The per-file diagnostics emitted by doclint are the actionable content; promoting them to the PR check panel and a clearly delimited summary block makes a doc-gen failure self-explanatory without leaving the PR. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? End-to-end on this branch with deliberately broken references planted in two code paths (mirroring the test pattern from PR #55605): - `ColumnarMap.java` (real Java source): `{link org.apache.spark.deliberately.NoSuchClass}` and `{link ColumnVector#nonExistentMethod()}`. - `Partition.scala` (Scala source via genjavadoc): `[[Partition.index]]` -- the `.`-separator case that javadoc treats as inner-class lookup. The Documentation generation job will fail with the expected `Fatal javadoc errors` summary block in the log and per-file inline annotations on this PR's check panel. The plant commit will be dropped before this PR is taken out of draft. The state machine was also exercised locally against a captured log from a prior failing doc-gen run; the captured fatal count matches javadoc's `N errors` summary line. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (Anthropic) Closes #55814 from cloud-fan/unidoc-fatal-summary. Authored-by: Wenchen Fan Signed-off-by: Wenchen Fan (cherry picked from commit 12b2595277e8dcafe6f1151744a24228dc04f701) Signed-off-by: Wenchen Fan --- docs/_plugins/build_api_docs.rb | 88 ++++++++++++++++++++++++++++++++- 1 file changed, 87 insertions(+), 1 deletion(-) diff --git a/docs/_plugins/build_api_docs.rb b/docs/_plugins/build_api_docs.rb index 1ef80bfaf09a4..429cef5aa026c 100644 --- a/docs/_plugins/build_api_docs.rb +++ b/docs/_plugins/build_api_docs.rb @@ -132,7 +132,7 @@ def build_spark_scala_and_java_docs_if_necessary command = "build/sbt -Pkinesis-asl unidoc" puts "Running '#{command}'..." - # Two filter passes on the unidoc output: + # Two filter passes on the unidoc output, plus an additive fatal-error summary: # # 1. Genjavadoc-stub diagnostic blocks (~28 `[error]` lines on stubs under # `target/java/`, plus 3-5 continuation lines each). Inert because @@ -146,6 +146,18 @@ def build_spark_scala_and_java_docs_if_necessary # per-file `error: reference not found` diagnostics) but carry no signal # of their own. Suppressing them brings the visible log from ~17K to ~5K # lines on a typical run while leaving every diagnostic untouched. + # + # 3. Fatal-error summary (additive, drops no log lines). The filtered log is + # still ~4K lines and most `error:` text in it is non-fatal source-loading + # chatter, so the build-failing diagnostics are hard to spot. After the + # pipe closes, we print a `Fatal javadoc errors (N): ...` block and emit + # `::error file=,line=::` GitHub Actions annotations so they surface in the + # PR check panel. Captured strictly within the Standard Doclet phase + # bracketed by `Building tree for all the packages and classes...` and + # `Building index for all classes...`, which is where doclint diagnostics + # are emitted -- this matches what javadoc counts toward exit code 1. + # Self-checked against javadoc's own `N errors` summary line; a mismatch + # emits a `::warning::` so future phase-marker drift is visible. ansi = /\e\[[0-9;]*[A-Za-z]/ stub_header = %r{ \[(?:error|warn)\]\s+ @@ -167,10 +179,51 @@ def build_spark_scala_and_java_docs_if_necessary |Generating\s+\S+\.html ) }x + + # Doclint phase tracking for the trailing summary. Standard Doclet bookends the + # phase that produces build-failing diagnostics with these marker lines; any + # `error:` outside this window is source-loading noise that does not contribute + # to javadoc's exit code. The summary below captures only the fatal ones and + # re-emits them as GitHub Actions annotations so they surface in the PR check + # panel instead of being buried in a 4K-line log. + doclint_start = %r{\bBuilding\s+tree\s+for\s+all\s+the\s+packages\s+and\s+classes\b} + doclint_end = %r{\bBuilding\s+index\s+for\s+all\s+classes\b} + doclint_diag = %r{\A\[warn\]\s+(?\S+):(?\d+)(?::\d+)?:\s+error:\s+(?.+?)\s*\z} + doclint_cont = %r{\A\[warn\]\s(?!\S+:\d+(?::\d+)?:\s+error:)(?.*?)\s*\z} + doclint_summary = %r{\A\[warn\]\s+(?[\d,]+)\s+errors?\s*\z} + in_stub = false + in_doclint = false + fatal_diagnostics = [] + pending_context_lines = 0 # snippet + caret lines that follow each diag header + reported_error_count = nil + IO.popen("#{command} 2>&1", 'r') do |pipe| pipe.each_line do |line| plain = line.gsub(ansi, '') + + if plain =~ doclint_start + in_doclint = true + elsif in_doclint && plain =~ doclint_end + in_doclint = false + pending_context_lines = 0 + end + + if in_doclint && (m = plain.match(doclint_diag)) + fatal_diagnostics << { + path: m[:path], line: m[:lineno], msg: m[:msg], context: [] + } + pending_context_lines = 2 + elsif in_doclint && pending_context_lines > 0 && + (m = plain.match(doclint_cont)) && !fatal_diagnostics.empty? + fatal_diagnostics.last[:context] << m[:content] + pending_context_lines -= 1 + end + + if reported_error_count.nil? && (m = plain.match(doclint_summary)) + reported_error_count = m[:count].delete(',').to_i + end + if plain =~ verbose_line in_stub = false # suppress -verbose progress line @@ -185,6 +238,39 @@ def build_spark_scala_and_java_docs_if_necessary end end end + + unless fatal_diagnostics.empty? + bar = "=" * 72 + puts "" + puts bar + puts "Fatal javadoc errors (#{fatal_diagnostics.size}):" + puts bar + fatal_diagnostics.each_with_index do |d, i| + puts " #{i + 1}. #{d[:path]}:#{d[:line]}: #{d[:msg]}" + d[:context].each { |c| puts " #{c}" } + end + puts bar + puts "" + + # GitHub Actions inline annotations. `%`, `\r`, `\n` require URL-style + # escaping per the workflow command spec; newlines render as multiple + # lines inside the annotation, so the source snippet and caret display + # under the error message in the PR check panel. + project_root = SPARK_PROJECT_ROOT + '/' + fatal_diagnostics.each do |d| + rel = d[:path].start_with?(project_root) ? d[:path][project_root.length..] : d[:path] + full = ([d[:msg]] + d[:context]).join("\n") + enc = full.gsub(/[%\r\n]/, '%' => '%25', "\r" => '%0D', "\n" => '%0A') + puts "::error file=#{rel},line=#{d[:line]},title=javadoc::#{enc}" + end + end + + if reported_error_count && reported_error_count != fatal_diagnostics.size + puts "::warning::Javadoc reported #{reported_error_count} errors but " \ + "build_api_docs.rb captured #{fatal_diagnostics.size}. The doclint " \ + "phase markers may have shifted; please update build_api_docs.rb." + end + raise("Unidoc generation failed") unless $?.success? end From 1c99423d81fdbfa443fcb0a9521611563c713bb0 Mon Sep 17 00:00:00 2001 From: Milan Dankovic Date: Wed, 13 May 2026 20:19:35 +0800 Subject: [PATCH 114/286] [SPARK-54735][SQL][REVERT] Revert column comments preservation in view with SCHEMA EVOLUTION This reverts commit e886428cdfd3f59232bc037e8665b70e75863d25. ### What changes were proposed in this pull request? This PR reverts the previous change that prevented user-defined comments in schema evolution views from being overwritten by comments from the underlying table. ### Why are the changes needed? The original change introduced a behavioral change in how comments are handled for views with SCHEMA EVOLUTION enabled. After further discussion, we concluded that some users may already depend on the existing behavior where comments are refreshed from the underlying table during schema evolution. Because this can be considered a breaking change, and given the proximity to the release cut, we decided to revert the original PR for now to preserve backward compatibility and avoid potential regressions for existing workloads. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55849 from miland-db/milan-dankovic_data/revert-SPARK-54735. Authored-by: Milan Dankovic Signed-off-by: Wenchen Fan (cherry picked from commit 965056eb99a90f9b52f701e6615f7df53ad01be5) Signed-off-by: Wenchen Fan --- .../apache/spark/sql/internal/SQLConf.scala | 14 -- .../sql/execution/datasources/rules.scala | 46 +---- .../results/view-schema-evolution.sql.out | 4 +- .../sql/execution/SQLViewTestSuite.scala | 159 +----------------- .../SparkConfigBindingPolicySuite.scala | 2 +- 5 files changed, 8 insertions(+), 217 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 9bb5b98cfabf6..fcb736e404854 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -2319,17 +2319,6 @@ object SQLConf { .booleanConf .createWithDefault(true) - val VIEW_SCHEMA_EVOLUTION_PRESERVE_USER_COMMENTS = - buildConf("spark.sql.view.schemaEvolution.preserveUserComments") - .internal() - .doc("When enabled, views with SCHEMA EVOLUTION mode will preserve user-set view comments " + - "when the underlying table schema evolves. When disabled, view comments will be " + - "overwritten with table comments on every schema sync.") - .version("4.2.0") - .withBindingPolicy(ConfigBindingPolicy.SESSION) - .booleanConf - .createWithDefault(true) - val OUTPUT_COMMITTER_CLASS = buildConf("spark.sql.sources.outputCommitterClass") .version("1.4.0") .internal() @@ -8056,9 +8045,6 @@ class SQLConf extends Serializable with Logging with SqlApiConf { def viewSchemaCompensation: Boolean = getConf(VIEW_SCHEMA_COMPENSATION) - def viewSchemaEvolutionPreserveUserComments: Boolean = - getConf(VIEW_SCHEMA_EVOLUTION_PRESERVE_USER_COMMENTS) - def defaultCacheStorageLevel: StorageLevel = StorageLevel.fromString(getConf(DEFAULT_CACHE_STORAGE_LEVEL).name()) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala index ff6c1e067406d..ce41bbe4aeb3d 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala @@ -40,7 +40,7 @@ import org.apache.spark.sql.execution.datasources.{CreateTable => CreateTableV1} import org.apache.spark.sql.execution.datasources.v2.FileDataSourceV2 import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.sources.InsertableRelation -import org.apache.spark.sql.types.{ArrayType, DataType, MapType, MetadataBuilder, StructField, StructType} +import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructField, StructType} import org.apache.spark.sql.util.PartitioningUtils.normalizePartitionSpec import org.apache.spark.sql.util.SchemaUtils import org.apache.spark.util.ArrayImplicits._ @@ -737,19 +737,6 @@ case class QualifyLocationWithWarehouse(catalog: SessionCatalog) extends Rule[Lo * It does so by walking the resolved plan looking for View operators for persisted views. */ object ViewSyncSchemaToMetaStore extends (LogicalPlan => Unit) { - - /** - * Checks if comment changes between view and table should trigger schema sync. - * When preserveUserComments flag is enabled, comment differences should NOT trigger sync - * because we want to preserve user-set view comments. - */ - private def shouldTriggerRedoOnCommentChange( - viewField: StructField, - tableField: StructField, - preserveUserComments: Boolean): Boolean = { - !preserveUserComments && viewField.getComment() != tableField.getComment() - } - def apply(plan: LogicalPlan): Unit = { plan.foreach { case View(metaData, false, viewQuery, _) @@ -768,44 +755,19 @@ object ViewSyncSchemaToMetaStore extends (LogicalPlan => Unit) { (field.dataType != planField.dataType || field.nullable != planField.nullable || (viewSchemaMode == SchemaEvolution && ( - field.name != planField.name || - shouldTriggerRedoOnCommentChange( - field, - planField, - session.sessionState.conf.viewSchemaEvolutionPreserveUserComments)))) + field.getComment() != planField.getComment() || + field.name != planField.name))) } - lazy val viewFieldsByName = viewFields.map(f => f.name -> f).toMap - if (redo) { val newSchema = if (viewSchemaMode == SchemaTypeEvolution) { val newFields = viewQuery.schema.map { case StructField(name, dataType, nullable, _) => StructField(name, dataType, nullable, - viewFieldsByName(name).metadata) - } - StructType(newFields) - } else if (session.sessionState.conf.viewSchemaEvolutionPreserveUserComments) { - // Adopt types/nullable/names from query, but preserve view comments. - val newFields = viewQuery.schema.map { planField => - val newMetadata = viewFieldsByName.get(planField.name) match { - case Some(viewField) => - // Use table metadata but override with view comment - val builder = new MetadataBuilder().withMetadata(planField.metadata) - viewField.getComment() match { - case Some(comment) => builder.putString("comment", comment) - case None => builder.remove("comment") - } - builder.build() - case None => - // New column, use table metadata as-is - planField.metadata - } - StructField(planField.name, planField.dataType, planField.nullable, newMetadata) + viewFields.find(_.name == name).get.metadata) } StructType(newFields) } else { - // Legacy behavior: adopt everything from table including comments. viewQuery.schema } SchemaUtils.checkColumnNameDuplication(fieldNames.toImmutableArraySeq, diff --git a/sql/core/src/test/resources/sql-tests/results/view-schema-evolution.sql.out b/sql/core/src/test/resources/sql-tests/results/view-schema-evolution.sql.out index 497e307f592fb..7410e7eaafd6f 100644 --- a/sql/core/src/test/resources/sql-tests/results/view-schema-evolution.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/view-schema-evolution.sql.out @@ -897,8 +897,8 @@ DESCRIBE EXTENDED v -- !query schema struct -- !query output -c1 bigint c1 6d -c2 string c2 6d +c1 bigint c1 6e +c2 string c2 6e # Detailed Table Information Catalog spark_catalog diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala index f55d4b8cb9e61..1e5b1cdfaeee7 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala @@ -33,7 +33,7 @@ import org.apache.spark.sql.connector.catalog.CatalogManager.SESSION_CATALOG_NAM import org.apache.spark.sql.errors.DataTypeErrors.toSQLId import org.apache.spark.sql.internal.SQLConf._ import org.apache.spark.sql.test.SharedSparkSession -import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType} +import org.apache.spark.sql.types.{IntegerType, StructField, StructType} import org.apache.spark.util.ArrayImplicits._ /** @@ -881,163 +881,6 @@ class PersistedViewTestSuite extends SQLViewTestSuite with SharedSparkSession { } } - test("Schema evolution views should preserve manually set comments") { - withTable("t") { - withView("v") { - // Create table with comments. - sql("CREATE TABLE t (c1 INT COMMENT " + - "'table comment 1', c2 STRING COMMENT 'table comment 2')") - sql("INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c')") - - // Create view with schema evolution (no column list) - initially adopts table comments. - sql("CREATE VIEW v WITH SCHEMA EVOLUTION AS SELECT * FROM t") - - // Verify initial comments from table are adopted. - val descInitial = sql("DESCRIBE EXTENDED v").collect() - val c1CommentInitial = descInitial.filter(r => r.getString(0) == "c1") - val c2CommentInitial = descInitial.filter(r => r.getString(0) == "c2") - assert(c1CommentInitial.nonEmpty && c1CommentInitial(0).getString(2) == "table comment 1", - "Initial c1 comment should be 'table comment 1' from table") - assert(c2CommentInitial.nonEmpty && c2CommentInitial(0).getString(2) == "table comment 2", - "Initial c2 comment should be 'table comment 2' from table") - - // Simulate user manually changing view comments (via UI or ALTER COLUMN). - val catalog = spark.sessionState.catalog - val viewMeta = catalog.getTableMetadata(TableIdentifier("v")) - val newSchema = StructType(Seq( - StructField("c1", IntegerType, nullable = true).withComment("user comment 1"), - StructField("c2", StringType, nullable = true).withComment("user comment 2") - )) - catalog.alterTable(viewMeta.copy(schema = newSchema)) - - // Verify manually set comments. - val descManual = sql("DESCRIBE EXTENDED v").collect() - val c1CommentManual = descManual.filter(r => r.getString(0) == "c1") - val c2CommentManual = descManual.filter(r => r.getString(0) == "c2") - assert(c1CommentManual.nonEmpty && c1CommentManual(0).getString(2) == "user comment 1", - "c1 comment should be 'user comment 1'") - assert(c2CommentManual.nonEmpty && c2CommentManual(0).getString(2) == "user comment 2", - "c2 comment should be 'user comment 2'") - - // SELECT from view (triggers ViewSyncSchemaToMetaStore). - checkAnswer(sql("SELECT * FROM v"), Seq(Row(1, "a"), Row(2, "b"), Row(3, "c"))) - - // Verify manually set comments are PRESERVED (not reverted to table comments). - val descAfterSelect = sql("DESCRIBE EXTENDED v").collect() - val c1CommentAfter = descAfterSelect.filter(r => r.getString(0) == "c1") - val c2CommentAfter = descAfterSelect.filter(r => r.getString(0) == "c2") - assert(c1CommentAfter.nonEmpty && c1CommentAfter(0).getString(2) == "user comment 1", - "c1 comment should still be 'user comment 1' after SELECT (bug: was reverted)") - assert(c2CommentAfter.nonEmpty && c2CommentAfter(0).getString(2) == "user comment 2", - "c2 comment should still be 'user comment 2' after SELECT (bug: was reverted)") - - // Verify that type changes are still adopted. - sql("DROP TABLE t") - sql("CREATE TABLE t (c1 BIGINT COMMENT 'table comment changed', " + - "c2 DOUBLE COMMENT 'table comment changed 2')") - sql("INSERT INTO t VALUES (4, 5.0), (6, 7.0)") - - // SELECT from view - should adopt new types but preserve view comments. - checkAnswer(sql("SELECT * FROM v"), Seq(Row(4, 5.0), Row(6, 7.0))) - - // Verify types changed but comments preserved. - val descAfterTypeChange = sql("DESCRIBE EXTENDED v").collect() - val c1Final = descAfterTypeChange.filter(r => r.getString(0) == "c1") - val c2Final = descAfterTypeChange.filter(r => r.getString(0) == "c2") - assert(c1Final.nonEmpty && c1Final(0).getString(1) == "bigint", - "c1 type should be updated to bigint") - assert(c2Final.nonEmpty && c2Final(0).getString(1) == "double", - "c2 type should be updated to double") - assert(c1Final.nonEmpty && c1Final(0).getString(2) == "user comment 1", - "c1 comment should still be 'user comment 1' (preserved)") - assert(c2Final.nonEmpty && c2Final(0).getString(2) == "user comment 2", - "c2 comment should still be 'user comment 2' (preserved)") - } - } - } - - test("Schema evolution comments legacy behavior with preserveUserComments=false") { - withSQLConf(VIEW_SCHEMA_EVOLUTION_PRESERVE_USER_COMMENTS.key -> "false") { - withTable("t") { - withView("v") { - // Create table with comments. - sql("CREATE TABLE t (c1 INT COMMENT " + - "'table comment 1', c2 STRING COMMENT 'table comment 2')") - sql("INSERT INTO t VALUES (1, 'a'), (2, 'b')") - - // Create view with schema evolution. - sql("CREATE VIEW v WITH SCHEMA EVOLUTION AS SELECT * FROM t") - - // User manually changes view comments. - val catalog = spark.sessionState.catalog - val viewMeta = catalog.getTableMetadata(TableIdentifier("v")) - val newSchema = StructType(Seq( - StructField("c1", IntegerType, nullable = true).withComment("user comment 1"), - StructField("c2", StringType, nullable = true).withComment("user comment 2") - )) - catalog.alterTable(viewMeta.copy(schema = newSchema)) - - // Verify manually set comments. - val descManual = sql("DESCRIBE EXTENDED v").collect() - val c1CommentManual = descManual.filter(r => r.getString(0) == "c1") - val c2CommentManual = descManual.filter(r => r.getString(0) == "c2") - assert(c1CommentManual.nonEmpty && c1CommentManual(0).getString(2) == "user comment 1") - assert(c2CommentManual.nonEmpty && c2CommentManual(0).getString(2) == "user comment 2") - - // SELECT from view (triggers ViewSyncSchemaToMetaStore). - checkAnswer(sql("SELECT * FROM v"), Seq(Row(1, "a"), Row(2, "b"))) - - // With flag=false, comments should REVERT to table comments (legacy behavior). - val descAfterSelect = sql("DESCRIBE EXTENDED v").collect() - val c1CommentAfter = descAfterSelect.filter(r => r.getString(0) == "c1") - val c2CommentAfter = descAfterSelect.filter(r => r.getString(0) == "c2") - assert(c1CommentAfter.nonEmpty && c1CommentAfter(0).getString(2) == "table comment 1", - "c1 comment should revert to 'table comment 1' (legacy behavior)") - assert(c2CommentAfter.nonEmpty && c2CommentAfter(0).getString(2) == "table comment 2", - "c2 comment should revert to 'table comment 2' (legacy behavior)") - } - } - } - } - - test("Comments are preserved when table comment changes with preserveUserComments=true") { - withTable("t") { - withView("v") { - // Create table with initial comment. - sql("CREATE TABLE t (c1 INT COMMENT 'original table comment')") - sql("INSERT INTO t VALUES (1), (2)") - - // Create view with schema evolution - inherits table comment. - sql("CREATE VIEW v WITH SCHEMA EVOLUTION AS SELECT * FROM t") - - // Verify view has inherited the table comment. - val descInitial = sql("DESCRIBE EXTENDED v").collect() - val c1Initial = descInitial.filter(r => r.getString(0) == "c1") - assert(c1Initial.nonEmpty && c1Initial(0).getString(2) == "original table comment", - "View should inherit table comment") - - // Change the table comment. - sql("ALTER TABLE t CHANGE COLUMN c1 c1 INT COMMENT 'new table comment'") - - // Verify table comment changed. - val descTable = sql("DESCRIBE EXTENDED t").collect() - val c1Table = descTable.filter(r => r.getString(0) == "c1") - assert(c1Table.nonEmpty && c1Table(0).getString(2) == "new table comment", - "Table comment should be updated") - - // SELECT from view (triggers ViewSyncSchemaToMetaStore). - checkAnswer(sql("SELECT * FROM v"), Seq(Row(1), Row(2))) - - // Verify view still has the original inherited comment (frozen). - val descAfterSelect = sql("DESCRIBE EXTENDED v").collect() - val c1AfterSelect = descAfterSelect.filter(r => r.getString(0) == "c1") - assert(c1AfterSelect.nonEmpty && - c1AfterSelect(0).getString(2) == "original table comment", - "View should preserve inherited comment even when table comment changes") - } - } - } - def getShowCreateDDL(view: String, serde: Boolean = false): String = { val result = if (serde) { sql(s"SHOW CREATE TABLE $view AS SERDE") diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala index 7b04db0788bd9..e7b4669e2b969 100644 --- a/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala +++ b/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala @@ -36,7 +36,7 @@ class SparkConfigBindingPolicySuite extends SparkFunSuite { test("Test adding bindingPolicy to config") { val allConfigs = SQLConf.getConfigEntries().asScala.filter { entry => - entry.key == SQLConf.VIEW_SCHEMA_EVOLUTION_PRESERVE_USER_COMMENTS.key + entry.key == SQLConf.PLAN_CHANGE_LOG_LEVEL.key } assert(allConfigs.head.bindingPolicy.isDefined) assert(allConfigs.head.bindingPolicy.get == ConfigBindingPolicy.SESSION) From 4ecc83138eea4ef7ac0a32483971597966cc6432 Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Wed, 13 May 2026 21:17:48 +0800 Subject: [PATCH 115/286] [SPARK-56811][UI] Restore sub-execution grouping on the SQL tab listing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? This PR restores sub-execution grouping on the SQL tab listing that was silently dropped when the listing was switched to server-side pagination in SPARK-56140. - Backend (`SqlResource.sqlTable`): accept a new `groupSubExecution` query parameter (default = cluster config `spark.ui.groupSQLSubExecutionEnabled`). When enabled, paginate over root executions only and embed each root's children as `subExecutions: [...]` on the row. Sub-executions whose root is missing from the filtered set surface as roots so they are never hidden. Fix `recordsTotal` / `recordsFiltered` to count root rows in grouped mode so the DataTables "Showing X to Y of Z entries" matches the visible rows. - Frontend (`allexecutionspage.js`): read the existing `group-sub-exec-config` data attribute, forward `groupSubExecution=true|false` to the server, and append a trailing **Sub Executions** column showing a `+N sub` toggle on roots that have children. Toggling expands the row using DataTables `row().child()` with a nested table — matches the SPARK-41752 / Spark 4.1 layout. - CSS (`webui-dataTables.css`): minimal styling for the toggle link and the nested child table (indent + tertiary background). ### Why are the changes needed? SPARK-41752 introduced sub-execution grouping in Spark 4.1, and SPARK-55875 carried it over when the listing moved to client-side DataTables. SPARK-56140 then switched the listing to server-side pagination but didn't carry over the grouping logic — every sub-execution now shows up as its own flat row, regressing the UX for queries such as `CACHE TABLE` and nested CTAS. ### Does this PR introduce _any_ user-facing change? Yes — the SQL tab listing again folds sub-executions under their root, with a `+N sub` toggle to expand. Default behaviour is controlled by the existing `spark.ui.groupSQLSubExecutionEnabled` config (default `true`). The screenshots below were taken from the same workload (`SELECT`, `CACHE TABLE`, broadcast join, `DROP TABLE`, `CREATE TABLE … AS`) — `CACHE TABLE` and the inner `CREATE TABLE` each spawn a sub-execution. **Before — flat listing (today, post-SPARK-56140):** sub-executions appear as flat sibling rows; e.g. id 2/3/4 are all `CACHE TABLE people_eng_cached` and id 7/8/9 are all `nested CTAS`. ![before](https://raw.githubusercontent.com/yaooqinn/spark/SPARK-56811-screenshots/screenshots/before-flat.png) **After — grouped, collapsed:** only roots are listed, with a trailing `+1 sub` toggle on rows that own a sub-execution. ![after-collapsed](https://raw.githubusercontent.com/yaooqinn/spark/SPARK-56811-screenshots/screenshots/after-collapsed.png) **After — grouped, expanded:** clicking `+1 sub` expands a nested table inline. ![after-expanded](https://raw.githubusercontent.com/yaooqinn/spark/SPARK-56811-screenshots/screenshots/after-expanded.png) ### How was this patch tested? - New unit test in `SqlResourceWithActualMetricsSuite` covering both grouped and flat modes against a session that runs `CACHE TABLE` (which produces a root + sub-execution pair). - Existing `SqlResourceWithActualMetricsSuite` and `AllExecutionsPageWithInMemoryStoreSuite` continue to pass. - Manual verification in a local Spark UI (screenshots above): ran the same workload twice — once with `spark.ui.groupSQLSubExecutionEnabled=false` (flat) and once with the default `true` (grouped) — and confirmed both modes render correctly. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot CLI 1.0.44-2 with Claude Opus 4.7 (Extra high reasoning) Closes #55787 from yaooqinn/SPARK-56811. Authored-by: Kent Yao Signed-off-by: Kent Yao (cherry picked from commit 6e4d9d7147afdc8d160a72ada9d57edbb86ec138) Signed-off-by: Kent Yao --- .../spark/ui/static/webui-dataTables.css | 32 +++ .../execution/ui/static/allexecutionspage.js | 229 +++++++++++++----- .../spark/status/api/v1/sql/SqlResource.scala | 96 ++++++-- .../SqlResourceWithActualMetricsSuite.scala | 114 +++++++++ 4 files changed, 384 insertions(+), 87 deletions(-) diff --git a/core/src/main/resources/org/apache/spark/ui/static/webui-dataTables.css b/core/src/main/resources/org/apache/spark/ui/static/webui-dataTables.css index 202579c6b67ce..e7a8f3ab0839a 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/webui-dataTables.css +++ b/core/src/main/resources/org/apache/spark/ui/static/webui-dataTables.css @@ -58,4 +58,36 @@ table.dataTable thead .sorting_desc_disabled::after { div.dataTables_wrapper div.dataTables_length select { width: 100%; +} + +/* SQL tab sub-execution disclosure (SPARK-56811) */ +table#sql-table td.sub-exec-toggle { + white-space: nowrap; +} + +table#sql-table td.sub-exec-toggle a.toggle-sub-exec { + text-decoration: none; +} + +table#sql-table td.sub-exec-toggle a.toggle-sub-exec:hover { + text-decoration: underline; +} + +table#sql-table tr.shown td.sub-exec-toggle a.toggle-sub-exec { + font-weight: 600; +} + +table#sql-table tr.shown + tr > td { + background-color: var(--bs-tertiary-bg, #f4f7fa); +} + +table.sub-exec-table { + margin-left: 1.5rem !important; + width: calc(100% - 1.5rem) !important; + background-color: transparent; +} + +table.sub-exec-table thead th { + font-weight: 600; + background-color: transparent; } \ No newline at end of file diff --git a/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js b/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js index 34e3be4913ce4..b741a18789d68 100644 --- a/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js +++ b/sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js @@ -116,6 +116,13 @@ $(document).ready(function () { } } + // Read the cluster-level grouping toggle rendered into the page by Scala + var groupSubExecEnabled = true; + var configEl = document.getElementById("group-sub-exec-config"); + if (configEl) { + groupSubExecEnabled = configEl.getAttribute("data-value") === "true"; + } + function init(resolvedAppId) { var sqlTableEndPoint = createSQLTableEndPoint(resolvedAppId); @@ -131,6 +138,94 @@ $(document).ready(function () { '
'; + var columns = [ + { + data: "id", name: "id", title: "ID", + render: function (data, type) { + if (type !== "display") return data; + var basePath = uiRoot + appBasePath; + return '' + + data + ''; + } + }, + { + data: "queryId", name: "queryId", title: "Query ID", + orderable: false, + render: function (data, type) { + if (type !== "display" || !data) return data || ""; + var safe = escapeHtml(data); + return '' + escapeHtml(data.substring(0, 8)) + '...'; + } + }, + { + data: "status", name: "status", title: "Status", + render: function (data, type) { + if (type !== "display") return data; + return statusBadge(data); + } + }, + { + data: "description", name: "description", title: "Description", + render: function (data, type, row) { + if (type !== "display") return data || ""; + return descriptionHtml({ id: row.id, description: data }); + } + }, + { + data: "submissionTime", name: "submissionTime", title: "Submitted", + render: function (data, type) { + if (type !== "display") return data; + return formatDateSql(data); + } + }, + { + data: "duration", name: "duration", title: "Duration", + render: function (data, type) { + if (type !== "display") return data; + return formatDurationSql(data); + } + }, + { + data: "jobIds", name: "jobIds", title: "Succeeded Jobs", + orderable: false, + render: function (data, type) { + if (type !== "display") return (data || []).join(","); + return jobIdLinks(data || []); + } + }, + { + data: "errorMessage", name: "errorMessage", title: "Error Message", + orderable: false, + render: function (data, type) { + if (type !== "display" || !data) return data || ""; + if (data.length > 100) { + return '' + + escapeHtml(data.substring(0, 100)) + '...'; + } + return escapeHtml(data); + } + } + ]; + if (groupSubExecEnabled) { + // Trailing "Sub Executions" column matching the SPARK-41752 / 4.1 layout: + // shows "+N sub" when the root has children, blank otherwise. Click to + // expand a child row containing the sub-execution rows. + columns.push({ + data: null, name: "subExecutions", title: "Sub Executions", + orderable: false, searchable: false, + className: "sub-exec-toggle", + render: function (data, type, row) { + if (type !== "display") return ""; + var subs = row.subExecutions || []; + if (subs.length === 0) return ""; + var childId = "sub-exec-" + row.id; + return '' + + '+' + subs.length + ' sub'; + } + }); + } + var table = $("#sql-table").DataTable({ serverSide: true, processing: true, @@ -146,83 +241,83 @@ $(document).ready(function () { if (sel) { d.status = sel; } + d.groupSubExecution = groupSubExecEnabled ? "true" : "false"; }, dataSrc: function (json) { return json.aaData; }, error: function () { $("#sql-table_processing").css("display", "none"); } }, - columns: [ - { - data: "id", name: "id", title: "ID", - render: function (data, type) { - if (type !== "display") return data; - var basePath = uiRoot + appBasePath; - return '' + - data + ''; - } - }, - { - data: "queryId", name: "queryId", title: "Query ID", - orderable: false, - render: function (data, type) { - if (type !== "display" || !data) return data || ""; - return '' + data.substring(0, 8) + '...'; - } - }, - { - data: "status", name: "status", title: "Status", - render: function (data, type) { - if (type !== "display") return data; - return statusBadge(data); - } - }, - { - data: "description", name: "description", title: "Description", - render: function (data, type, row) { - if (type !== "display") return data || ""; - return descriptionHtml({ id: row.id, description: data }); - } - }, - { - data: "submissionTime", name: "submissionTime", title: "Submitted", - render: function (data, type) { - if (type !== "display") return data; - return formatDateSql(data); - } - }, - { - data: "duration", name: "duration", title: "Duration", - render: function (data, type) { - if (type !== "display") return data; - return formatDurationSql(data); - } - }, - { - data: "jobIds", name: "jobIds", title: "Succeeded Jobs", - orderable: false, - render: function (data, type) { - if (type !== "display") return (data || []).join(","); - return jobIdLinks(data || []); - } - }, - { - data: "errorMessage", name: "errorMessage", title: "Error Message", - orderable: false, - render: function (data, type) { - if (type !== "display" || !data) return data || ""; - if (data.length > 100) { - return '' + - escapeHtml(data.substring(0, 100)) + '...'; - } - return escapeHtml(data); - } - } - ], + columns: columns, order: [[0, "desc"]], language: { search: "Search: " } }); + // Child-row expansion for sub-executions. Sub data is embedded per root row + // in the server payload (`row.subExecutions`), so no second fetch is needed. + // Under serverSide: true DataTables destroys/recreates rows on every sort, + // filter or page change, so we track expanded row IDs out-of-band and + // re-attach the child on each draw. + if (groupSubExecEnabled) { + var expandedRowIds = {}; + + var renderSubExecutionsHtml = function (rowData) { + var subs = (rowData && rowData.subExecutions) || []; + var basePath = uiRoot + appBasePath; + var childId = "sub-exec-" + (rowData && rowData.id); + var html = ''; + html += '' + + ''; + subs.forEach(function (child) { + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + }); + html += '
IDStatusDescriptionDurationSucceeded Jobs
' + child.id + '' + statusBadge(child.status) + '' + descriptionHtml({ + id: child.id, description: child.description || "" + }) + '' + formatDurationSql(child.duration) + '' + jobIdLinks(child.jobIds || []) + '
'; + return html; + }; + + $("#sql-table tbody").on("click", "a.toggle-sub-exec", function (e) { + e.preventDefault(); + var tr = $(this).closest("tr"); + var dtRow = table.row(tr); + var rowData = dtRow.data(); + var subs = (rowData && rowData.subExecutions) || []; + if (dtRow.child.isShown()) { + dtRow.child.hide(); + tr.removeClass("shown"); + $(this).text("+" + subs.length + " sub").attr("aria-expanded", "false"); + delete expandedRowIds[rowData.id]; + } else { + dtRow.child(renderSubExecutionsHtml(rowData)).show(); + tr.addClass("shown"); + $(this).text("\u2212" + subs.length + " sub").attr("aria-expanded", "true"); + expandedRowIds[rowData.id] = true; + } + }); + + table.on("draw", function () { + $("#sql-table tbody > tr").each(function () { + var dtRow = table.row(this); + var data = dtRow.data(); + if (data && expandedRowIds[data.id]) { + var subs = data.subExecutions || []; + dtRow.child(renderSubExecutionsHtml(data)).show(); + $(this).addClass("shown"); + $(this).find("a.toggle-sub-exec") + .text("\u2212" + subs.length + " sub") + .attr("aria-expanded", "true"); + } + }); + }); + } + $("#status-filter").on("change", function () { table.draw(); }); diff --git a/sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/SqlResource.scala b/sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/SqlResource.scala index 91d3f9a484e24..59a1264993e19 100644 --- a/sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/SqlResource.scala +++ b/sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/SqlResource.scala @@ -19,12 +19,14 @@ package org.apache.spark.status.api.v1.sql import java.util.{Date, HashMap} +import scala.jdk.CollectionConverters._ import scala.util.{Failure, Success, Try} import jakarta.ws.rs._ import jakarta.ws.rs.core.{Context, MediaType, UriInfo} import org.apache.spark.JobExecutionStatus +import org.apache.spark.internal.config.UI.UI_SQL_GROUP_SUB_EXECUTION_ENABLED import org.apache.spark.sql.execution.ui.{SparkPlanGraph, SparkPlanGraphCluster, SparkPlanGraphNode, SQLAppStatusStore, SQLExecutionUIData} import org.apache.spark.status.api.v1.{BaseAppResource, NotFoundException} import org.apache.spark.ui.UIUtils @@ -74,6 +76,11 @@ private[v1] class SqlResource extends BaseAppResource { * Server-side DataTables endpoint for SQL executions listing. * Accepts DataTables server-side parameters (start, length, order, search) * and returns paginated results with recordsTotal/recordsFiltered counts. + * + * When `groupSubExecution=true` (default = `spark.ui.groupSQLSubExecutionEnabled`), + * pagination is over root executions only and each row carries its sub-executions + * inline as `subExecutions: [...]`. Sub-executions whose root is missing from the + * filtered set (orphans) are surfaced as roots so they don't disappear. */ @GET @Path("sqlTable") @@ -85,7 +92,11 @@ private[v1] class SqlResource extends BaseAppResource { // Echo draw counter to prevent stale responses val draw = Option(uriParams.getFirst("draw")).map(_.toInt).getOrElse(0) - val totalRecords = sqlStore.executionsCount() + // Sub-execution grouping flag; default to the cluster config. Defensive + // parse - bad values should not 500 the public REST endpoint. + val groupSubExec = Option(uriParams.getFirst("groupSubExecution")) + .flatMap(v => Try(v.toBoolean).toOption) + .getOrElse(ui.conf.get(UI_SQL_GROUP_SUB_EXECUTION_ENABLED)) // Search and status filter val searchValue = Option(uriParams.getFirst("search[value]")) @@ -94,9 +105,14 @@ private[v1] class SqlResource extends BaseAppResource { .filter(_.nonEmpty) val needsFilter = searchValue.isDefined || statusFilter.isDefined + // Always load all execs once. We need the full set to (a) identify orphan + // sub-executions whose root is filtered out and (b) count root rows for + // `recordsTotal`. `sqlStore.executionsList()` is already a full + // materialization, so there is no separate "KVStore-pagination" path being + // disabled here. + val allExecs = sqlStore.executionsList() + val filteredExecs = if (needsFilter) { - // When filtering, we must load all and filter in memory - val allExecs = sqlStore.executionsList() allExecs.filter { exec => val matchesSearch = searchValue.forall { search => val lower = search.toLowerCase(java.util.Locale.ROOT) @@ -110,10 +126,14 @@ private[v1] class SqlResource extends BaseAppResource { matchesSearch && matchesStatus } } else { - // No filter — will use KVStore pagination below - Seq.empty + allExecs + } + + val (rootRows, subsByRoot) = if (groupSubExec) { + SqlResource.partitionRoots(filteredExecs) + } else { + (filteredExecs, Map.empty[Long, Seq[SQLExecutionUIData]]) } - val filteredRecords = if (needsFilter) filteredExecs.size else totalRecords // Sort val sortCol = Option(uriParams.getFirst("order[0][column]")) @@ -125,26 +145,43 @@ private[v1] class SqlResource extends BaseAppResource { val start = Option(uriParams.getFirst("start")).map(_.toInt).getOrElse(0) val length = Option(uriParams.getFirst("length")).map(_.toInt).getOrElse(20) - val page = if (needsFilter) { - // Filter/search: sort and paginate in memory - val sorted = sortExecs(filteredExecs, sortCol, sortDir) - if (length > 0) sorted.slice(start, start + length) else sorted - } else { - // No filter: use KVStore-level pagination for efficiency - // KVStore returns in insertion order; sort in memory for the page - val execs = sqlStore.executionsList() - val sorted = sortExecs(execs, sortCol, sortDir) - if (length > 0) sorted.slice(start, start + length) else sorted + val sortedRoots = sortExecs(rootRows, sortCol, sortDir) + val page = if (length > 0) sortedRoots.slice(start, start + length) else sortedRoots + + // Convert to Java-compatible row data; embed sub-executions when grouping. + // Always emit a `subExecutions` field (possibly empty) in grouped mode so + // JSON consumers see a consistent schema; flat mode never includes it. + val aaData = page.map { exec => + val row = execToRow(exec) + if (groupSubExec) { + val subs = subsByRoot.getOrElse(exec.executionId, Seq.empty) + // Sort subs by id ascending so they appear in chronological order + row.put("subExecutions", sortExecs(subs, "id", "asc").map(execToRow).asJava) + } + row } - // Convert to Java-compatible row data - val aaData = page.map(execToRow) + // Counts: grouped totals reflect root-only counts so DataTables shows + // "Showing X to Y of Z entries" matching the rows the user actually sees. + // Flat mode's recordsTotal is the unfiltered total (from the KVStore), + // which lets DataTables show the "filtered from W total entries" suffix. + val recordsTotal = if (groupSubExec) { + if (needsFilter) { + // Re-derive root rows from the unfiltered set using the same predicate + SqlResource.partitionRoots(allExecs)._1.size + } else { + rootRows.size + } + } else { + sqlStore.executionsCount() + } + val recordsFiltered = if (groupSubExec) rootRows.size else filteredExecs.size val ret = new HashMap[String, Object]() ret.put("draw", Integer.valueOf(draw)) ret.put("aaData", aaData) - ret.put("recordsTotal", java.lang.Long.valueOf(filteredRecords)) - ret.put("recordsFiltered", java.lang.Long.valueOf(filteredRecords)) + ret.put("recordsTotal", java.lang.Long.valueOf(recordsTotal)) + ret.put("recordsFiltered", java.lang.Long.valueOf(recordsFiltered)) ret } } @@ -275,3 +312,22 @@ private[v1] class SqlResource extends BaseAppResource { } } + +private[v1] object SqlResource { + + /** + * Split a set of executions into root rows and a sub-execution map. A root row is + * either an execution whose id equals its rootExecutionId, or an orphan sub whose + * root parent is absent from the input set. Called on the filtered set (for paging) + * and on the full set (for `recordsTotal`), so the predicate lives in one place + * rather than being inlined twice. + */ + def partitionRoots(execs: Seq[SQLExecutionUIData]) + : (Seq[SQLExecutionUIData], Map[Long, Seq[SQLExecutionUIData]]) = { + val ids = execs.iterator.map(_.executionId).toSet + val (roots, subs) = execs.partition { e => + e.executionId == e.rootExecutionId || !ids.contains(e.rootExecutionId) + } + (roots, subs.groupBy(_.rootExecutionId)) + } +} diff --git a/sql/core/src/test/scala/org/apache/spark/status/api/v1/sql/SqlResourceWithActualMetricsSuite.scala b/sql/core/src/test/scala/org/apache/spark/status/api/v1/sql/SqlResourceWithActualMetricsSuite.scala index e375678157b2f..d6a5008f57494 100644 --- a/sql/core/src/test/scala/org/apache/spark/status/api/v1/sql/SqlResourceWithActualMetricsSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/status/api/v1/sql/SqlResourceWithActualMetricsSuite.scala @@ -32,6 +32,7 @@ import org.apache.spark.sql.DataFrame import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException import org.apache.spark.sql.catalyst.plans.SQLHelper import org.apache.spark.sql.execution.metric.SQLMetricsTestUtils +import org.apache.spark.sql.execution.ui.SQLExecutionUIData import org.apache.spark.sql.internal.SQLConf.ADAPTIVE_EXECUTION_ENABLED import org.apache.spark.sql.test.SharedSparkSession @@ -216,6 +217,119 @@ class SqlResourceWithActualMetricsSuite } } + test("SPARK-56811: sqlTable groups sub-executions under their root execution") { + // CACHE TABLE produces a root execution plus an inner sub-execution that + // shares its rootExecutionId. This is the canonical case where the SQL + // listing should fold the sub row under the root rather than flattening it. + spark.sql("CREATE OR REPLACE TEMP VIEW spark_56811 AS SELECT id FROM RANGE(10)") + .collect() + spark.sql("CACHE TABLE spark_56811_cached AS SELECT * FROM spark_56811").collect() + try { + eventually(timeout(10.seconds), interval(1.second)) { + val baseUrl = spark.sparkContext.ui.get.webUrl + + s"/api/v1/applications/${spark.sparkContext.applicationId}/sql/sqlTable" + + // Grouping ON: roots only, with subExecutions embedded on the root that + // owns a sub-execution. + val groupedUrl = new URI( + s"$baseUrl?start=0&length=100&draw=1&groupSubExecution=true").toURL + val (groupedCode, groupedOpt, _) = getContentAndCode(groupedUrl) + assert(groupedCode === HttpServletResponse.SC_OK) + val groupedJson = JsonMethods.parse(groupedOpt.get) + val groupedRecordsTotal = (groupedJson \ "recordsTotal").extract[Long] + val groupedRecordsFiltered = (groupedJson \ "recordsFiltered").extract[Long] + val groupedRows = (groupedJson \ "aaData").children + assert(groupedRecordsTotal === groupedRows.size, + "with no filter, recordsTotal should match returned root count") + assert(groupedRecordsFiltered === groupedRows.size, + "with no filter, recordsFiltered should match returned root count") + // Every row in grouped mode is either a true root (id == rootExecutionId) + // or an orphan sub whose real parent is absent from the result set. + val visibleIds = groupedRows.map(r => (r \ "id").extract[Long]).toSet + groupedRows.foreach { row => + val id = (row \ "id").extract[Long] + val rootId = (row \ "rootExecutionId").extract[Long] + assert(id == rootId || !visibleIds.contains(rootId), + s"grouped row $id (rootId=$rootId) is neither a root nor an orphan") + } + val rootsWithSubs = groupedRows.filter { row => + (row \ "subExecutions").children.nonEmpty + } + assert(rootsWithSubs.nonEmpty, + "CACHE TABLE should produce at least one root with sub-executions") + rootsWithSubs.foreach { row => + val rootId = (row \ "id").extract[Long] + (row \ "subExecutions").children.foreach { sub => + assert((sub \ "rootExecutionId").extract[Long] === rootId, + "sub-execution should reference its parent root") + assert((sub \ "id").extract[Long] !== rootId, + "sub-execution must not have the same id as its root") + } + } + + // Grouping OFF: flat list of every execution, with no embedded subs. + val flatUrl = new URI( + s"$baseUrl?start=0&length=100&draw=2&groupSubExecution=false").toURL + val (flatCode, flatOpt, _) = getContentAndCode(flatUrl) + assert(flatCode === HttpServletResponse.SC_OK) + val flatJson = JsonMethods.parse(flatOpt.get) + val flatRows = (flatJson \ "aaData").children + assert(flatRows.size > groupedRows.size, + "flat listing should contain at least one extra sub-execution row") + val embeddedSubs = groupedRows.map(r => (r \ "subExecutions").children.size).sum + assert(flatRows.size === groupedRows.size + embeddedSubs, + "flat size should equal grouped roots plus embedded sub rows") + flatRows.foreach { row => + assert((row \ "subExecutions").children.isEmpty, + "flat listing should not embed subExecutions") + } + } + } finally { + spark.sql("UNCACHE TABLE IF EXISTS spark_56811_cached") + } + } + + test("SPARK-56811: partitionRoots surfaces orphan sub-executions as root rows") { + def mkExec(id: Long, rootId: Long): SQLExecutionUIData = new SQLExecutionUIData( + executionId = id, + rootExecutionId = rootId, + description = s"exec $id", + details = "", + physicalPlanDescription = "", + modifiedConfigs = Map.empty, + metrics = Seq.empty, + submissionTime = id, + completionTime = None, + errorMessage = None, + jobs = Map.empty, + stages = Set.empty, + metricValues = null, + queryId = null) + + // Tree: + // 1 (root) -> 2, 3 (subs) + // 4 (root, no subs) + // 6 (sub of 5, but 5 is missing -> orphan) + val root1 = mkExec(1, 1) + val sub2 = mkExec(2, 1) + val sub3 = mkExec(3, 1) + val root4 = mkExec(4, 4) + val orphan6 = mkExec(6, 5) + + val (roots, subsByRoot) = + SqlResource.partitionRoots(Seq(root1, sub2, sub3, root4, orphan6)) + + assert(roots.map(_.executionId).toSet === Set(1L, 4L, 6L), + "true roots and orphan subs should both be promoted to root rows") + assert(subsByRoot.keySet === Set(1L), + "only execs with a parent present in the input should appear in subsByRoot") + assert(subsByRoot(1L).map(_.executionId).toSet === Set(2L, 3L), + "subs should be grouped under their parent root id") + val orphanRow = roots.find(_.executionId == 6L).get + assert(orphanRow.rootExecutionId === 5L, + "orphan promoted to a root row preserves its original rootExecutionId") + } + test("SPARK-56137: sqlList returns ISO date format in submissionTime") { withSQLConf(ADAPTIVE_EXECUTION_ENABLED.key -> "false") { spark.sql("SELECT 'date_format_test'").collect() From c70c91b5fc59b27c543d9731080eb1be7a9509df Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Wed, 13 May 2026 07:59:12 -0700 Subject: [PATCH 116/286] [SPARK-56843][BUILD] Update `checkJavaVersion` to ban 25.0.[0-2] properly ### What changes were proposed in this pull request? Require Java 25.0.3 or later in `checkJavaVersion` function when building on Java 25 in order to avoid JDK-8377811 - https://bugs.openjdk.org/browse/JDK-8377811 (G1GC: Optional Evacuations may evacuate pinned objects) This is consistent with our documentation too. - https://github.com/apache/spark/pull/55798 https://github.com/apache/spark/blob/42422741167664dc0d2ad535279b2a4c67906539/docs/index.md?plain=1#L38 ### Why are the changes needed? The current check only enforces `java.minimum.version=17.0.11` and treats any JDK with a higher feature version as compatible, so early Java 25 updates (25.0.0/0.1/0.2) silently pass. Pinning Java 25 builds to 25.0.3+ avoids those updates without affecting Java 17/21 users. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual build with `build/sbt core/compile` on Java 17.0.11 and 21.x (pass), and confirmed the new error message is thrown when `currentVersionFeature == 25 && currentVersionUpdate < 3`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 Closes #55840 from dongjoon-hyun/SPARK-56843. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun (cherry picked from commit 3963aba37618f4b47e1dea0960628d7ead69f157) Signed-off-by: Dongjoon Hyun --- project/SparkBuild.scala | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index 49556aa1df182..6b89a0e6ba9f5 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -125,6 +125,11 @@ object SparkBuild extends PomBuild { "The Java version used to build the project is outdated. " + s"Please use Java $minimumVersion or later.") } + if (currentVersionFeature == 25 && currentVersionUpdate < 3) { + throw new MessageOnlyException( + s"Java 25 requires update 3 or later due to JDK-8377811. " + + s"Current version: $currentVersion. Please use Java 25.0.3 or later.") + } }, (Compile / compile) := ((Compile / compile) dependsOn checkJavaVersion).value, (Test / compile) := ((Test / compile) dependsOn checkJavaVersion).value From b5d5e41b47f2b13d8900141aafc809035a2d1bbe Mon Sep 17 00:00:00 2001 From: Ziya Mukhtarov Date: Wed, 13 May 2026 10:39:09 -0700 Subject: [PATCH 117/286] [SPARK-56680][SQL] DSv2 INSERT and Insert-Only MERGE Metrics ### What changes were proposed in this pull request? MERGE and INSERT / write metrics for DSv2. This PR only handles pure-inserts. Overwrites will be handled in a future PR, as those require some values from the connector. ### Why are the changes needed? For better visibility into what happened as a result of DML queries. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added metric verification to existing tests. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 Closes #55586 from ZiyaZa/insert-only-merge-metrics. Lead-authored-by: Ziya Mukhtarov Co-authored-by: Gengliang Wang Signed-off-by: Gengliang Wang (cherry picked from commit c4d16a5bf0bd1b9dcac831454f869744854da79e) Signed-off-by: Gengliang Wang --- .../sql/connector/write/InsertSummary.java | 34 ++++++ .../analysis/RewriteMergeIntoTable.scala | 6 +- .../catalyst/plans/logical/v2Commands.scala | 20 ++++ .../connector/write/InsertSummaryImpl.scala | 24 ++++ .../connector/catalog/InMemoryBaseTable.scala | 27 +++-- .../InMemoryRowLevelOperationTable.scala | 20 +--- .../sql/connector/catalog/InMemoryTable.scala | 3 +- .../catalog/InMemoryTableWithV2Filter.scala | 3 +- .../datasources/v2/DataSourceV2Strategy.scala | 25 ++++- .../datasources/v2/TableCapabilityCheck.scala | 5 +- .../execution/datasources/v2/V2Writes.scala | 9 +- .../v2/WriteToDataSourceV2Exec.scala | 74 +++++++++--- ...SourceV2DataFrameSessionCatalogSuite.scala | 7 +- .../DataSourceV2DataFrameSuite.scala | 17 +++ .../sql/connector/DataSourceV2SQLSuite.scala | 1 + .../spark/sql/connector/InsertIntoTests.scala | 106 ++++++++++++++++++ .../connector/MergeIntoTableSuiteBase.scala | 30 +++++ .../sql/connector/V1WriteFallbackSuite.scala | 3 + 18 files changed, 364 insertions(+), 50 deletions(-) create mode 100644 sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/InsertSummary.java create mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/connector/write/InsertSummaryImpl.scala diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/InsertSummary.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/InsertSummary.java new file mode 100644 index 0000000000000..40f41bf238447 --- /dev/null +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/InsertSummary.java @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector.write; + +import org.apache.spark.annotation.Evolving; + +/** + * Provides an informational summary of the INSERT operation producing write. + * + * @since 4.2.0 + */ +@Evolving +public interface InsertSummary extends WriteSummary { + + /** + * Returns the number of inserted rows, or -1 if not found. + */ + long numInsertedRows(); +} diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala index 168f30623ee4c..8281f89bd2e8e 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala @@ -22,7 +22,7 @@ import org.apache.spark.sql.catalyst.expressions.{Alias, And, Attribute, Attribu import org.apache.spark.sql.catalyst.expressions.Literal.{FalseLiteral, TrueLiteral} import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression import org.apache.spark.sql.catalyst.plans.{FullOuter, Inner, JoinType, LeftAnti, LeftOuter, RightOuter} -import org.apache.spark.sql.catalyst.plans.logical.{AppendData, DeleteAction, Filter, HintInfo, InsertAction, Join, JoinHint, LogicalPlan, MergeAction, MergeIntoTable, MergeRows, NO_BROADCAST_AND_REPLICATION, Project, ReplaceData, UpdateAction, WriteDelta} +import org.apache.spark.sql.catalyst.plans.logical.{DeleteAction, Filter, HintInfo, InsertAction, InsertOnlyMerge, Join, JoinHint, LogicalPlan, MergeAction, MergeIntoTable, MergeRows, NO_BROADCAST_AND_REPLICATION, Project, ReplaceData, UpdateAction, WriteDelta} import org.apache.spark.sql.catalyst.plans.logical.MergeRows.{Copy, Delete, Discard, Insert, Instruction, Keep, ROW_ID, Split, Update} import org.apache.spark.sql.catalyst.util.RowDeltaUtils.{COPY_OPERATION, INSERT_OPERATION, OPERATION_COLUMN, UPDATE_OPERATION} import org.apache.spark.sql.connector.catalog.SupportsRowLevelOperations @@ -73,7 +73,7 @@ object RewriteMergeIntoTable extends RewriteRowLevelCommand with PredicateHelper } val project = Project(projectList, joinPlan) - AppendData.byPosition(r, project) + InsertOnlyMerge(r, project) case _ => m @@ -114,7 +114,7 @@ object RewriteMergeIntoTable extends RewriteRowLevelCommand with PredicateHelper output = generateExpandOutput(r.output, outputs), joinPlan) - AppendData.byPosition(r, mergeRows) + InsertOnlyMerge(r, mergeRows) case _ => m diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala index 7b657ce34df45..b1ab46ee94817 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala @@ -194,6 +194,26 @@ object AppendData { } } +/** + * Append data to an existing table as the result of an insert-only MERGE rewrite. + * + * Functionally equivalent to [[AppendData]] but distinguishes the row-level MERGE rewrite path. + */ +case class InsertOnlyMerge( + table: NamedRelation, + query: LogicalPlan, + write: Option[Write] = None, + analyzedQuery: Option[LogicalPlan] = None) extends V2WriteCommand with TransactionalWrite { + override val isByName: Boolean = false + override val withSchemaEvolution: Boolean = false + override val writePrivileges: Set[TableWritePrivilege] = Set(TableWritePrivilege.INSERT) + override def withNewQuery(newQuery: LogicalPlan): InsertOnlyMerge = copy(query = newQuery) + override def withNewTable(newTable: NamedRelation): InsertOnlyMerge = copy(table = newTable) + override def storeAnalyzedQuery(): Command = copy(analyzedQuery = Some(query)) + override protected def withNewChildInternal(newChild: LogicalPlan): InsertOnlyMerge = + copy(query = newChild) +} + /** * Overwrite data matching a filter in an existing table. */ diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/write/InsertSummaryImpl.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/write/InsertSummaryImpl.scala new file mode 100644 index 0000000000000..97c2e082c2573 --- /dev/null +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/write/InsertSummaryImpl.scala @@ -0,0 +1,24 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector.write + +/** + * Implementation of [[InsertSummary]] that provides INSERT operation summary. + */ +private[sql] case class InsertSummaryImpl(numInsertedRows: Long) extends InsertSummary { +} diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala index 53e57153030b7..f582f3e408cb6 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala @@ -788,30 +788,43 @@ abstract class InMemoryBaseTable( } override def abort(messages: Array[WriterCommitMessage]): Unit = {} + + protected def doCommit(messages: Array[WriterCommitMessage]): Unit + + override final def commit(messages: Array[WriterCommitMessage]): Unit = { + doCommit(messages) + commits += Commit(Instant.now().toEpochMilli) + } + + override final def commit( + messages: Array[WriterCommitMessage], + summary: WriteSummary): Unit = { + doCommit(messages) + commits += Commit(Instant.now().toEpochMilli, writeSummary = Some(summary)) + } } class Append(val info: LogicalWriteInfo) extends TestBatchWrite { - - override def commit(messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { + override protected def doCommit( + messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { withData(messages.map(_.asInstanceOf[BufferedRows])) - commits += Commit(Instant.now().toEpochMilli) } } class DynamicOverwrite(val info: LogicalWriteInfo) extends TestBatchWrite { - override def commit(messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { + override protected def doCommit( + messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { val newData = messages.map(_.asInstanceOf[BufferedRows]) dataMap --= newData.flatMap(_.rows.map(getKey)) withData(newData) - commits += Commit(Instant.now().toEpochMilli) } } class TruncateAndAppend(val info: LogicalWriteInfo) extends TestBatchWrite { - override def commit(messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { + override protected def doCommit( + messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { dataMap.clear() withData(messages.map(_.asInstanceOf[BufferedRows])) - commits += Commit(Instant.now().toEpochMilli) } } diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTable.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTable.scala index 6778f4489e459..5c0bc0b143f3d 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTable.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryRowLevelOperationTable.scala @@ -17,7 +17,6 @@ package org.apache.spark.sql.connector.catalog -import java.time.Instant import java.util import org.apache.spark.sql.catalyst.InternalRow @@ -26,7 +25,7 @@ import org.apache.spark.sql.connector.catalog.constraints.Constraint import org.apache.spark.sql.connector.distributions.{Distribution, Distributions} import org.apache.spark.sql.connector.expressions.{FieldReference, LogicalExpressions, NamedReference, SortDirection, SortOrder, Transform} import org.apache.spark.sql.connector.read.{Scan, ScanBuilder} -import org.apache.spark.sql.connector.write.{BatchWrite, DeltaBatchWrite, DeltaWrite, DeltaWriteBuilder, DeltaWriter, DeltaWriterFactory, LogicalWriteInfo, PhysicalWriteInfo, RequiresDistributionAndOrdering, RowLevelOperation, RowLevelOperationBuilder, RowLevelOperationInfo, SupportsDelta, Write, WriteBuilder, WriterCommitMessage, WriteSummary} +import org.apache.spark.sql.connector.write.{BatchWrite, DeltaBatchWrite, DeltaWrite, DeltaWriteBuilder, DeltaWriter, DeltaWriterFactory, LogicalWriteInfo, PhysicalWriteInfo, RequiresDistributionAndOrdering, RowLevelOperation, RowLevelOperationBuilder, RowLevelOperationInfo, SupportsDelta, Write, WriteBuilder, WriterCommitMessage} import org.apache.spark.sql.connector.write.RowLevelOperation.Command import org.apache.spark.sql.types.StructType import org.apache.spark.sql.util.CaseInsensitiveStringMap @@ -143,18 +142,11 @@ class InMemoryRowLevelOperationTable private ( override def description(): String = "InMemoryPartitionReplaceOperation" } - abstract class RowLevelOperationBatchWrite extends TestBatchWrite { - - override def commit(messages: Array[WriterCommitMessage], metrics: WriteSummary): Unit = { - commit(messages) - commits += Commit(Instant.now().toEpochMilli, Some(metrics)) - } - } - private case class PartitionBasedReplaceData(scan: InMemoryBatchScan) - extends RowLevelOperationBatchWrite { + extends TestBatchWrite { - override def commit(messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { + override protected def doCommit( + messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { val newData = messages.map(_.asInstanceOf[BufferedRows]) val readRows = scan.data.flatMap(_.asInstanceOf[BufferedRows].rows) val readPartitions = readRows.map(r => getKey(r, schema)).distinct @@ -216,12 +208,12 @@ class InMemoryRowLevelOperationTable private ( } } - private object TestDeltaBatchWrite extends RowLevelOperationBatchWrite with DeltaBatchWrite{ + private object TestDeltaBatchWrite extends TestBatchWrite with DeltaBatchWrite { override def createBatchWriterFactory(info: PhysicalWriteInfo): DeltaWriterFactory = { new DeltaBufferedRowsWriterFactory(CatalogV2Util.v2ColumnsToStructType(columns())) } - override def commit(messages: Array[WriterCommitMessage]): Unit = { + override protected def doCommit(messages: Array[WriterCommitMessage]): Unit = { val newData = messages.map(_.asInstanceOf[BufferedRows]) withDeletes(newData) withData(newData, columns()) diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala index 15ed4136dbda8..66db9c18fa981 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala @@ -205,7 +205,8 @@ class InMemoryTable( private class Overwrite(filters: Array[Filter]) extends TestBatchWrite { import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.MultipartIdentifierHelper - override def commit(messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { + override protected def doCommit( + messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { val deleteKeys = InMemoryTable.filtersToKeys( dataMap.keys, partCols.map(_.toSeq.quoted).toImmutableArraySeq, filters) dataMap --= deleteKeys diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithV2Filter.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithV2Filter.scala index f2827faf59435..e9d73d0f9fe1e 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithV2Filter.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithV2Filter.scala @@ -140,7 +140,8 @@ class InMemoryTableWithV2Filter( private class Overwrite(predicates: Array[Predicate]) extends TestBatchWrite { import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.MultipartIdentifierHelper - override def commit(messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { + override protected def doCommit( + messages: Array[WriterCommitMessage]): Unit = dataMap.synchronized { val deleteKeys = InMemoryTableWithV2Filter.filtersToKeys( dataMap.keys, partCols.map(_.toSeq.quoted).toImmutableArraySeq, predicates) dataMap --= deleteKeys diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala index e113475811092..3c58298ec9211 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala @@ -24,7 +24,7 @@ import org.apache.hadoop.fs.Path import org.apache.spark.{SparkException, SparkIllegalArgumentException} import org.apache.spark.internal.Logging import org.apache.spark.internal.LogKeys.EXPR -import org.apache.spark.sql.catalyst.analysis.{ResolvedIdentifier, ResolvedNamespace, ResolvedPartitionSpec, ResolvedPersistentView, ResolvedTable, ResolvedTempView} +import org.apache.spark.sql.catalyst.analysis.{NamedRelation, ResolvedIdentifier, ResolvedNamespace, ResolvedPartitionSpec, ResolvedPersistentView, ResolvedTable, ResolvedTempView} import org.apache.spark.sql.catalyst.catalog.CatalogUtils import org.apache.spark.sql.catalyst.expressions import org.apache.spark.sql.catalyst.expressions.{And, Attribute, DynamicPruning, Expression, NamedExpression, Not, Or, PredicateHelper, SubqueryExpression} @@ -41,7 +41,7 @@ import org.apache.spark.sql.connector.expressions.{FieldReference, LiteralValue} import org.apache.spark.sql.connector.expressions.filter.{And => V2And, Not => V2Not, Or => V2Or, Predicate} import org.apache.spark.sql.connector.read.LocalScan import org.apache.spark.sql.connector.read.streaming.{ContinuousStream, MicroBatchStream, SupportsRealTimeMode} -import org.apache.spark.sql.connector.write.V1Write +import org.apache.spark.sql.connector.write.{V1Write, Write} import org.apache.spark.sql.errors.{QueryCompilationErrors, QueryExecutionErrors} import org.apache.spark.sql.execution.{FilterExec, InSubqueryExec, LeafExecNode, LocalTableScanExec, ProjectExec, RowDataSourceScanExec, ScalarSubquery => ExecScalarSubquery, SparkPlan, SparkStrategy => Strategy} import org.apache.spark.sql.execution.command.{CommandUtils, CreateMetricViewCommand, MetricViewHelper} @@ -491,8 +491,8 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat invalidateCache) :: Nil } - case AppendData(r @ ExtractV2Table(v1: SupportsWrite), _, _, - _, _, Some(write), analyzedQuery) if v1.supports(TableCapability.V1_BATCH_WRITE) => + case AppendWrite(r @ ExtractV2Table(v1: SupportsWrite), Some(write), analyzedQuery) + if v1.supports(TableCapability.V1_BATCH_WRITE) => write match { case v1Write: V1Write => assert(analyzedQuery.isDefined) @@ -505,6 +505,9 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat case AppendData(r: DataSourceV2Relation, query, _, _, _, Some(write), _) => AppendDataExec(planLater(query), refreshCache(r), write, r.name) :: Nil + case InsertOnlyMerge(r: DataSourceV2Relation, query, Some(write), _) => + InsertOnlyMergeExec(planLater(query), refreshCache(r), write, r.name) :: Nil + case OverwriteByExpression(r @ ExtractV2Table(v1: SupportsWrite), _, _, _, _, _, Some(write), analyzedQuery) if v1.supports(TableCapability.V1_BATCH_WRITE) => write match { @@ -843,6 +846,20 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat } } +/** + * Pattern that matches either an [[AppendData]] or an [[InsertOnlyMerge]] and exposes the + * fields needed to plan the v1 batch-write fallback path. + */ +private object AppendWrite { + def unapply( + plan: LogicalPlan + ): Option[(NamedRelation, Option[Write], Option[LogicalPlan])] = plan match { + case a: AppendData => Some((a.table, a.write, a.analyzedQuery)) + case m: InsertOnlyMerge => Some((m.table, m.write, m.analyzedQuery)) + case _ => None + } +} + private[sql] object DataSourceV2Strategy extends Logging { private def translateLeafNodeFilterV2(predicate: Expression): Option[Predicate] = { diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableCapabilityCheck.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableCapabilityCheck.scala index b25059bd7bac1..988aa86db1d34 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableCapabilityCheck.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableCapabilityCheck.scala @@ -18,7 +18,7 @@ package org.apache.spark.sql.execution.datasources.v2 import org.apache.spark.sql.catalyst.expressions.Literal -import org.apache.spark.sql.catalyst.plans.logical.{AppendData, LogicalPlan, OverwriteByExpression, OverwritePartitionsDynamic} +import org.apache.spark.sql.catalyst.plans.logical.{AppendData, InsertOnlyMerge, LogicalPlan, OverwriteByExpression, OverwritePartitionsDynamic} import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2 import org.apache.spark.sql.connector.catalog.Table import org.apache.spark.sql.connector.catalog.TableCapability._ @@ -49,6 +49,9 @@ object TableCapabilityCheck extends (LogicalPlan => Unit) { case AppendData(r: DataSourceV2Relation, _, _, _, _, _, _) if !supportsBatchWrite(r.table) => throw QueryCompilationErrors.unsupportedAppendInBatchModeError(r.name) + case InsertOnlyMerge(r: DataSourceV2Relation, _, _, _) if !supportsBatchWrite(r.table) => + throw QueryCompilationErrors.unsupportedAppendInBatchModeError(r.name) + case OverwritePartitionsDynamic(r: DataSourceV2Relation, _, _, _, _, _) if !r.table.supports(BATCH_WRITE) || !r.table.supports(OVERWRITE_DYNAMIC) => throw QueryCompilationErrors.unsupportedDynamicOverwriteInBatchModeError(r.table) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala index 70a2bf6d5b8b9..0cbf260457ffb 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala @@ -22,7 +22,7 @@ import java.util.UUID import scala.jdk.CollectionConverters._ import org.apache.spark.sql.catalyst.expressions.PredicateHelper -import org.apache.spark.sql.catalyst.plans.logical.{AppendData, LogicalPlan, OverwriteByExpression, OverwritePartitionsDynamic, ReplaceData, WriteDelta} +import org.apache.spark.sql.catalyst.plans.logical.{AppendData, InsertOnlyMerge, LogicalPlan, OverwriteByExpression, OverwritePartitionsDynamic, ReplaceData, WriteDelta} import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.catalyst.streaming.InternalOutputModes._ import org.apache.spark.sql.catalyst.util.WriteDeltaProjections @@ -51,6 +51,13 @@ object V2Writes extends Rule[LogicalPlan] with PredicateHelper { val newQuery = DistributionAndOrderingUtils.prepareQuery(write, query, r.funCatalog) a.copy(write = Some(write), query = newQuery) + case m @ InsertOnlyMerge(r: DataSourceV2Relation, query, None, _) => + val writeOptions = r.options.asCaseSensitiveMap.asScala.toMap + val writeBuilder = newWriteBuilder(r.table, writeOptions, query.schema) + val write = writeBuilder.build() + val newQuery = DistributionAndOrderingUtils.prepareQuery(write, query, r.funCatalog) + m.copy(write = Some(write), query = newQuery) + case o @ OverwriteByExpression( r: DataSourceV2Relation, deleteExpr, query, options, _, _, None, _) => // fail if any filter cannot be converted. correctness depends on removing all matching data. diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala index 3cbfed40d876e..5d8b5a081c900 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala @@ -33,16 +33,16 @@ import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Column, Identifier import org.apache.spark.sql.connector.catalog.transactions.Transaction import org.apache.spark.sql.connector.expressions.Transform import org.apache.spark.sql.connector.metric.CustomMetric -import org.apache.spark.sql.connector.write.{BatchWrite, DataWriter, DataWriterFactory, DeleteSummaryImpl, DeltaWrite, DeltaWriter, MergeSummaryImpl, PhysicalWriteInfoImpl, RowLevelOperation, RowLevelOperationTable, UpdateSummaryImpl, Write, WriterCommitMessage, WriteSummary} +import org.apache.spark.sql.connector.write.{BatchWrite, DataWriter, DataWriterFactory, DeleteSummaryImpl, DeltaWrite, DeltaWriter, InsertSummaryImpl, MergeSummaryImpl, PhysicalWriteInfoImpl, RowLevelOperation, RowLevelOperationTable, UpdateSummaryImpl, Write, WriterCommitMessage, WriteSummary} import org.apache.spark.sql.connector.write.RowLevelOperation.Command._ import org.apache.spark.sql.errors.{QueryCompilationErrors, QueryExecutionErrors} -import org.apache.spark.sql.execution.{QueryExecution, SparkPlan, UnaryExecNode} +import org.apache.spark.sql.execution.{QueryExecution, SparkPlan, SQLExecution, UnaryExecNode} import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper import org.apache.spark.sql.execution.metric.{CustomMetrics, SQLMetric, SQLMetrics} import org.apache.spark.sql.types.StructType import org.apache.spark.sql.util.SchemaValidationMode.PROHIBIT_CHANGES -import org.apache.spark.util.{LongAccumulator, Utils} import org.apache.spark.util.ArrayImplicits._ +import org.apache.spark.util.Utils /** * Deprecated logical plan for writing data into data source v2. This is being replaced by more @@ -294,6 +294,37 @@ case class AppendDataExec( override def withTransaction(txn: Option[Transaction]): AppendDataExec = copy(transaction = txn) override protected def withNewChildInternal(newChild: SparkPlan): AppendDataExec = copy(query = newChild) + + override protected def getWriteSummary(): Option[WriteSummary] = + Some(InsertSummaryImpl(numInsertedRows = numOutputRowsMetric.value)) +} + +/** + * Physical plan for an insert-only MERGE rewrite. Behaves like [[AppendDataExec]] but emits a + * [[org.apache.spark.sql.connector.write.MergeSummary]] so commit metadata reports the operation + * as a MERGE, with all output rows accounted for as inserts. + */ +case class InsertOnlyMergeExec( + query: SparkPlan, + refreshCache: () => Unit, + write: Write, + tableName: String, + transaction: Option[Transaction] = None) extends V2ExistingTableWriteExec { + override def withTransaction(txn: Option[Transaction]): InsertOnlyMergeExec = + copy(transaction = txn) + override protected def withNewChildInternal(newChild: SparkPlan): InsertOnlyMergeExec = + copy(query = newChild) + + override protected def getWriteSummary(): Option[WriteSummary] = + Some(MergeSummaryImpl( + numTargetRowsCopied = 0L, + numTargetRowsDeleted = 0L, + numTargetRowsUpdated = 0L, + numTargetRowsInserted = numOutputRowsMetric.value, + numTargetRowsMatchedUpdated = 0L, + numTargetRowsMatchedDeleted = 0L, + numTargetRowsNotMatchedBySourceUpdated = 0L, + numTargetRowsNotMatchedBySourceDeleted = 0L)) } /** @@ -477,17 +508,18 @@ trait V2ExistingTableWriteExec extends V2TableWriteExec with TransactionalExec { trait RowLevelWriteExec extends V2ExistingTableWriteExec { def rowLevelCommand: RowLevelOperation.Command - override protected lazy val sparkMetrics: Map[String, SQLMetric] = rowLevelCommand match { - case UPDATE => - Map( - "numUpdatedRows" -> SQLMetrics.createMetric(sparkContext, "number of updated rows"), - "numCopiedRows" -> SQLMetrics.createMetric(sparkContext, "number of copied rows")) - case DELETE => - Map( - "numDeletedRows" -> SQLMetrics.createMetric(sparkContext, "number of deleted rows"), - "numCopiedRows" -> SQLMetrics.createMetric(sparkContext, "number of copied rows")) - case _ => Map.empty - } + override protected lazy val sparkMetrics: Map[String, SQLMetric] = super.sparkMetrics ++ ( + rowLevelCommand match { + case UPDATE => + Map( + "numUpdatedRows" -> SQLMetrics.createMetric(sparkContext, "number of updated rows"), + "numCopiedRows" -> SQLMetrics.createMetric(sparkContext, "number of copied rows")) + case DELETE => + Map( + "numDeletedRows" -> SQLMetrics.createMetric(sparkContext, "number of deleted rows"), + "numCopiedRows" -> SQLMetrics.createMetric(sparkContext, "number of copied rows")) + case _ => Map.empty + }) /** * Returns the value of the named metric, or -1 if the metric is not found. @@ -551,6 +583,12 @@ trait V2TableWriteExec override def customMetrics: Map[String, SQLMetric] = Map.empty + protected lazy val numOutputRowsMetric: SQLMetric = + SQLMetrics.createMetric(sparkContext, "number of output rows") + + override protected def sparkMetrics: Map[String, SQLMetric] = Map( + "numOutputRows" -> numOutputRowsMetric) + protected def writeWithV2(batchWrite: BatchWrite): Seq[InternalRow] = { val rdd: RDD[InternalRow] = { val tempRdd = query.execute() @@ -568,7 +606,6 @@ trait V2TableWriteExec PhysicalWriteInfoImpl(rdd.getNumPartitions)) val useCommitCoordinator = batchWrite.useCommitCoordinator val messages = new Array[WriterCommitMessage](rdd.partitions.length) - val totalNumRowsAccumulator = new LongAccumulator() logInfo(log"Start processing data source write support: " + log"${MDC(LogKeys.BATCH_WRITE, batchWrite)}. The input RDD has " + @@ -586,11 +623,14 @@ trait V2TableWriteExec (index, result: DataWritingSparkTaskResult) => { val commitMessage = result.writerCommitMessage messages(index) = commitMessage - totalNumRowsAccumulator.add(result.numRows) + numOutputRowsMetric.add(result.numRows) batchWrite.onDataWriterCommit(commitMessage) } ) + val executionId = sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY) + SQLMetrics.postDriverMetricUpdates(sparkContext, executionId, Seq(numOutputRowsMetric)) + val writeSummary = getWriteSummary() logInfo(log"Data source write support ${MDC(LogKeys.BATCH_WRITE, batchWrite)} is committing.") writeSummary match { @@ -598,7 +638,7 @@ trait V2TableWriteExec case None => batchWrite.commit(messages) } logInfo(log"Data source write support ${MDC(LogKeys.BATCH_WRITE, batchWrite)} committed.") - commitProgress = Some(StreamWriterCommitProgress(totalNumRowsAccumulator.value)) + commitProgress = Some(StreamWriterCommitProgress(numOutputRowsMetric.value)) } catch { case cause: Throwable => logError( diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala index 4db59b36c1fec..97cdebe2d32df 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala @@ -181,7 +181,7 @@ object InMemoryTableSessionCatalog { private [connector] trait SessionCatalogTest[T <: Table, Catalog <: TestV2SessionCatalogBase[T]] extends SharedSparkSession - with BeforeAndAfter { + with BeforeAndAfter { self: InsertIntoSQLOnlyTests => protected def catalog(name: String): CatalogPlugin = { spark.sessionState.catalogManager.catalog(name) @@ -215,6 +215,7 @@ private [connector] trait SessionCatalogTest[T <: Table, Catalog <: TestV2Sessio val t1 = "tbl" val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") df.write.format(v2Format).saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, df) } @@ -222,6 +223,7 @@ private [connector] trait SessionCatalogTest[T <: Table, Catalog <: TestV2Sessio val t1 = "tbl" val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") df.write.format(v2Format).mode("append").saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, df) } @@ -245,10 +247,12 @@ private [connector] trait SessionCatalogTest[T <: Table, Catalog <: TestV2Sessio df.select("id", "data").write.format(v2Format).saveAsTable(t1) } df.write.format(v2Format).mode("append").saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, df) // Check that appends are by name df.select($"data", $"id").write.format(v2Format).mode("append").saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, df.union(df)) } @@ -284,6 +288,7 @@ private [connector] trait SessionCatalogTest[T <: Table, Catalog <: TestV2Sessio val t1 = "tbl" val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") df.write.format(v2Format).mode("ignore").saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, df) } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala index 5cda5169369e7..c532ef359a7c1 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala @@ -102,7 +102,9 @@ class DataSourceV2DataFrameSuite sql(s"CREATE TABLE $t2 (id bigint, data string) USING foo") val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") df.write.insertInto(t1) + checkInsertMetrics(t1, numInsertedRows = 3) spark.table(t1).write.insertInto(t2) + checkInsertMetrics(t2, numInsertedRows = 3) checkAnswer(spark.table(t2), df) } } @@ -112,6 +114,7 @@ class DataSourceV2DataFrameSuite withTable(t1) { val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") df.write.saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) checkAnswer(spark.table(t1), df) } } @@ -129,6 +132,7 @@ class DataSourceV2DataFrameSuite // appends are by name not by position df.select($"data", $"id").write.mode("append").saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) checkAnswer(spark.table(t1), df) } } @@ -157,6 +161,7 @@ class DataSourceV2DataFrameSuite withTable(t1) { val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") df.write.mode("ignore").saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) checkAnswer(spark.table(t1), df) } } @@ -190,6 +195,7 @@ class DataSourceV2DataFrameSuite val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") df.write.option("other", "20").mode("append").saveAsTable(t1) + checkInsertMetrics(t1, numInsertedRows = 3) sparkContext.listenerBus.waitUntilEmpty() plan match { @@ -391,24 +397,29 @@ class DataSourceV2DataFrameSuite val df1 = Seq((1, "hr")).toDF("id", "dep") df1.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 1) sql(s"ALTER TABLE $tableName ADD COLUMN txt STRING DEFAULT 'initial-text'") val df2 = Seq((2, "hr"), (3, "software")).toDF("id", "dep") df2.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 2) sql(s"ALTER TABLE $tableName ALTER COLUMN txt SET DEFAULT 'new-text'") val df3 = Seq((4, "hr"), (5, "hr")).toDF("id", "dep") df3.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 2) val df4 = Seq((6, "hr", null), (7, "hr", "explicit-text")).toDF("id", "dep", "txt") df4.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 2) sql(s"ALTER TABLE $tableName ALTER COLUMN txt DROP DEFAULT") val df5 = Seq((8, "hr"), (9, "hr")).toDF("id", "dep") df5.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 2) checkAnswer( sql(s"SELECT * FROM $tableName"), @@ -432,11 +443,13 @@ class DataSourceV2DataFrameSuite val df1 = Seq(1, 2).toDF("id") df1.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 2) sql(s"ALTER TABLE $tableName ALTER COLUMN dep SET DEFAULT 'it'") val df2 = Seq(3, 4).toDF("id") df2.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 2) checkAnswer( sql(s"SELECT * FROM $tableName"), @@ -450,6 +463,7 @@ class DataSourceV2DataFrameSuite val df3 = Seq(1, 2).toDF("id") df3.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 2) checkAnswer( sql(s"SELECT * FROM $tableName"), @@ -493,11 +507,13 @@ class DataSourceV2DataFrameSuite val df1 = Seq(1).toDF("id") df1.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 1) sql(s"ALTER TABLE $tableName ALTER COLUMN dep SET DEFAULT ('i' || 't')") val df2 = Seq(2).toDF("id") df2.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $tableName"), @@ -536,6 +552,7 @@ class DataSourceV2DataFrameSuite val df3 = Seq(1).toDF("id") df3.writeTo(tableName).append() + checkInsertMetrics(tableName, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $tableName"), diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala index e49cb692e3b3c..cb7531a0dbafd 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala @@ -439,6 +439,7 @@ class DataSourceV2SQLSuiteV1Filter Seq((basicCatalog, basicIdentifier), (atomicCatalog, atomicIdentifier)).foreach { case (catalog, identifier) => spark.sql(s"CREATE TABLE $identifier USING foo AS SELECT id, data FROM source") + checkInsertMetrics(identifier, numInsertedRows = 3) val table = catalog.loadTable(Identifier.of(Array(), "table_name")) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala index fa6edc96ec9fd..4f023136a6fe1 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala @@ -24,6 +24,9 @@ import org.apache.spark.sql._ import org.apache.spark.sql.catalyst.parser.CatalystSqlParser import org.apache.spark.sql.catalyst.util.TypeUtils._ import org.apache.spark.sql.catalyst.util.quoteIdentifier +import org.apache.spark.sql.connector.catalog.InMemoryBaseTable +import org.apache.spark.sql.connector.write.InsertSummary +import org.apache.spark.sql.execution.datasources.v2.ExtractV2Table import org.apache.spark.sql.functions.{array, lit, map, struct} import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.internal.SQLConf.{PARTITION_OVERWRITE_MODE, PartitionOverwriteMode} @@ -60,6 +63,7 @@ abstract class InsertIntoTests( sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") doInsert(t1, df) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, df) } @@ -70,6 +74,7 @@ abstract class InsertIntoTests( val dfr = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("data", "id") doInsert(t1, dfr) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, df) } @@ -79,6 +84,7 @@ abstract class InsertIntoTests( sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") doInsert(t1, df) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, df) } } @@ -89,6 +95,7 @@ abstract class InsertIntoTests( val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") val df2 = Seq((4L, "d"), (5L, "e"), (6L, "f")).toDF("id", "data") doInsert(t1, df) + checkInsertMetrics(t1, numInsertedRows = 3) doInsert(t1, df2, SaveMode.Overwrite) verifyTable(t1, df2) } @@ -99,6 +106,7 @@ abstract class InsertIntoTests( sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") val init = Seq((2L, "dummy"), (4L, "keep")).toDF("id", "data") doInsert(t1, init) + checkInsertMetrics(t1, numInsertedRows = 2) val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data") doInsert(t1, df, SaveMode.Overwrite) @@ -114,6 +122,7 @@ abstract class InsertIntoTests( sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") val init = Seq((2L, "dummy"), (4L, "keep")).toDF("id", "data") doInsert(t1, init) + checkInsertMetrics(t1, numInsertedRows = 2) val dfr = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("data", "id") doInsert(t1, dfr, SaveMode.Overwrite) @@ -210,6 +219,15 @@ trait InsertIntoSQLOnlyTests /** Check that the results in `tableName` match the `expected` DataFrame. */ protected def verifyTable(tableName: String, expected: DataFrame): Unit + protected def checkInsertMetrics(tableName: String, numInsertedRows: Long): Unit = { + val inMemoryTable = spark.table(tableName).queryExecution.analyzed.collectFirst { + case ExtractV2Table(t) => t.asInstanceOf[InMemoryBaseTable] + }.get + val summary = inMemoryTable.commits.last.writeSummary.get.asInstanceOf[InsertSummary] + assert(summary.numInsertedRows() === numInsertedRows, + s"Expected numInsertedRows=$numInsertedRows, got ${summary.numInsertedRows()}") + } + protected val v2Format: String protected val catalogAndNamespace: String @@ -293,6 +311,7 @@ trait InsertIntoSQLOnlyTests withTableAndData(t1) { view => sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") sql(s"INSERT INTO $t1 PARTITION (id = 23) SELECT data FROM $view") + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, sql(s"SELECT 23, data FROM $view")) } } @@ -303,6 +322,7 @@ trait InsertIntoSQLOnlyTests withTableAndData(t1) { view => sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy'), (4L, 'also-deleted')") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (id) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a"), @@ -317,6 +337,7 @@ trait InsertIntoSQLOnlyTests withTableAndData(t1) { view => sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy'), (4L, 'keep')") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (id) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a"), @@ -332,6 +353,7 @@ trait InsertIntoSQLOnlyTests withTableAndData(t1) { view => sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy'), (4L, 'also-deleted')") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 SELECT * FROM $view") verifyTable(t1, Seq( (1, "a"), @@ -346,6 +368,7 @@ trait InsertIntoSQLOnlyTests withTableAndData(t1) { view => sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy'), (4L, 'keep')") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 SELECT * FROM $view") verifyTable(t1, Seq( (1, "a"), @@ -361,6 +384,7 @@ trait InsertIntoSQLOnlyTests sql(s"CREATE TABLE $t1 (id bigint, data string, p1 int) " + s"USING $v2Format PARTITIONED BY (p1)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy', 23), (4L, 'keep', 2)") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (p1 = 23) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a", 23), @@ -377,6 +401,7 @@ trait InsertIntoSQLOnlyTests sql(s"CREATE TABLE $t1 (id bigint, data string, p int) " + s"USING $v2Format PARTITIONED BY (id, p)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy', 2), (4L, 'also-deleted', 2)") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (id, p = 2) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a", 2), @@ -393,6 +418,7 @@ trait InsertIntoSQLOnlyTests sql(s"CREATE TABLE $t1 (id bigint, data string, p int) " + s"USING $v2Format PARTITIONED BY (id, p)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy', 2), (4L, 'also-deleted', 2)") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (p = 2, id) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a", 2), @@ -409,6 +435,7 @@ trait InsertIntoSQLOnlyTests sql(s"CREATE TABLE $t1 (id bigint, data string, p int) " + s"USING $v2Format PARTITIONED BY (id, p)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy', 2), (4L, 'also-deleted', 2)") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (p = 2) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a", 2), @@ -424,6 +451,7 @@ trait InsertIntoSQLOnlyTests sql(s"CREATE TABLE $t1 (id bigint, data string, p int) " + s"USING $v2Format PARTITIONED BY (id, p)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy', 2), (4L, 'keep', 2)") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (p = 2, id) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a", 2), @@ -439,6 +467,7 @@ trait InsertIntoSQLOnlyTests sql(s"CREATE TABLE $t1 (id bigint, data string, p int) " + s"USING $v2Format PARTITIONED BY (id, p)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy', 2), (4L, 'keep', 2)") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (id, p = 2) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a", 2), @@ -454,6 +483,7 @@ trait InsertIntoSQLOnlyTests sql(s"CREATE TABLE $t1 (id bigint, data string, p int) " + s"USING $v2Format PARTITIONED BY (id, p)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy', 2), (4L, 'keep', 2)") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (p = 2) SELECT * FROM $view") verifyTable(t1, Seq( (1, "a", 2), @@ -469,6 +499,7 @@ trait InsertIntoSQLOnlyTests sql(s"CREATE TABLE $t1 (id bigint, data string, p int) " + s"USING $v2Format PARTITIONED BY (id, p)") sql(s"INSERT INTO $t1 VALUES (2L, 'dummy', 2), (4L, 'keep', 2)") + checkInsertMetrics(t1, numInsertedRows = 2) sql(s"INSERT OVERWRITE TABLE $t1 PARTITION (id = 2, p = 2) SELECT data FROM $view") verifyTable(t1, Seq( (2, "a", 2), @@ -491,6 +522,7 @@ trait InsertIntoSQLOnlyTests df.where("true").take(5) df.where("true").tail(5) + checkInsertMetrics(t1, numInsertedRows = 3) verifyTable(t1, spark.table(view)) } } @@ -510,9 +542,11 @@ trait InsertIntoSQLOnlyTests withTable(t1) { sql(s"CREATE TABLE $t1 (c1 INT DEFAULT 42, c2 STRING DEFAULT 'hello') USING $v2Format") sql(s"INSERT INTO $t1 VALUES (1, DEFAULT)") + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer(sql(s"SELECT * FROM $t1"), Row(1, "hello")) sql(s"INSERT INTO $t1 VALUES (DEFAULT, DEFAULT)") + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1 ORDER BY c1"), Seq(Row(1, "hello"), Row(42, "hello"))) @@ -565,8 +599,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format " + s"TBLPROPERTIES ('auto-schema-evolution' = 'false')") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) // Same column count, no evolution needed: should succeed even without capability. doInsertWithSchemaEvolution(t1, Seq((2L, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) verifyTable(t1, Seq((1L, "a"), (2L, "b")).toDF("id", "data")) } } @@ -576,7 +612,9 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "b")).toDF("x", "y")) + checkInsertMetrics(t1, numInsertedRows = 1) // No evolution verifyTable(t1, Seq((1L, "a"), (2L, "b")).toDF("id", "data")) } @@ -587,8 +625,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "b", true)).toDF("id", "data", "active")) + checkInsertMetrics(t1, numInsertedRows = 1) verifyTable(t1, Seq[(Long, String, java.lang.Boolean)]( (1L, "a", null), (2L, "b", true) @@ -601,8 +641,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "b", true, 100L)).toDF("id", "data", "active", "score")) + checkInsertMetrics(t1, numInsertedRows = 1) verifyTable(t1, Seq[(Long, String, java.lang.Boolean, java.lang.Long)]( (1L, "a", null, null), (2L, "b", true, 100L) @@ -615,7 +657,9 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "b", true)).toDF("x", "y", "z")) + checkInsertMetrics(t1, numInsertedRows = 1) verifyTable(t1, Seq[(Long, String, java.lang.Boolean)]( (1L, "a", null), (2L, "b", true) @@ -629,6 +673,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsertWithSchemaEvolution(t1, Seq((1L, "a", true)).toDF("id", "data", "active")) + checkInsertMetrics(t1, numInsertedRows = 1) verifyTable(t1, Seq( (1L, "a", true) ).toDF("id", "data", "active")) @@ -642,9 +687,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1, "Alice")).toDF("id", "name") .select($"id", struct($"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2, "Bob", 30)).toDF("id", "name", "age") .select($"id", struct($"name", $"age").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1, Row("Alice", null)), Row(2, Row("Bob", 30)))) @@ -658,9 +705,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1, "Alice")).toDF("id", "name") .select($"id", struct($"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2, "Bob", 30)).toDF("id", "firstName", "age") .select($"id", struct($"firstName", $"age").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1, Row("Alice", null)), Row(2, Row("Bob", 30)))) @@ -672,8 +721,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq(("b", true, 2L)).toDF("data", "active", "id"), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) verifyTable(t1, Seq[(Long, String, java.lang.Boolean)]( (1L, "a", null), (2L, "b", true) @@ -688,10 +739,12 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1, "Alice")).toDF("id", "name") .select($"id", struct($"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2, 30, "Bob")).toDF("id", "age", "name") .select($"id", struct($"age", $"name").as("info")), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1, Row("Alice", null)), Row(2, Row("Bob", 30)))) @@ -705,10 +758,12 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1, "Alice")).toDF("id", "name") .select($"id", struct($"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2, 30, "Bob", "NYC")).toDF("id", "age", "name", "city") .select($"id", struct($"age", $"name", $"city").as("info")), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1, Row("Alice", null, null)), Row(2, Row("Bob", 30, "NYC")))) @@ -720,8 +775,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq(("b", 2L)).toDF("data", "id"), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) // No evolution verifyTable(t1, Seq((1L, "a"), (2L, "b")).toDF("id", "data")) } @@ -732,8 +789,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq(("b", 2L)).toDF("x", "y"), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) verifyTable(t1, Seq[(java.lang.Long, String, String, java.lang.Long)]( (1L, "a", null, null), (null, null, "b", 2L) @@ -748,6 +807,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1, "Alice")).toDF("id", "name") .select($"id", struct($"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2, 30, "Bob")).toDF("id", "age", "name") .select($"id", struct($"age", $"name").as("info")), @@ -764,6 +824,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") doInsert(t1, Seq((1L, "a"), (2L, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 2) // REPLACE WHERE only deletes rows matching the predicate, then inserts new data. doInsertWithSchemaEvolution(t1, Seq((2L, "x", true), (4L, "y", false)).toDF("id", "data", "active"), @@ -781,6 +842,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") doInsert(t1, Seq((1L, "a"), (2L, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 2) doInsertWithSchemaEvolution(t1, Seq((true, "x", 2L), (false, "y", 4L)).toDF("active", "data", "id"), byName = true, @@ -801,6 +863,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => val initDf = Seq((1L, "Alice"), (2L, "Bob")).toDF("id", "name") .select($"id", struct($"name").as("info")) doInsert(t1, initDf) + checkInsertMetrics(t1, numInsertedRows = 2) doInsertWithSchemaEvolution(t1, Seq((2L, "Bobby", 25)).toDF("id", "name", "age") .select($"id", struct($"name", $"age").as("info")), @@ -820,6 +883,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => val initDf = Seq((1L, "Alice"), (2L, "Bob")).toDF("id", "name") .select($"id", struct($"name").as("info")) doInsert(t1, initDf) + checkInsertMetrics(t1, numInsertedRows = 2) doInsertWithSchemaEvolution(t1, Seq((2L, "Bobby", 25)).toDF("id", "name", "age") .select($"id", struct($"age", $"name").as("info")), @@ -853,6 +917,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") doInsert(t1, Seq((1L, "a"), (2L, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 2) // Overwrite with schema evolution adding a new column, dynamic mode should only replace // partitions present in the inserted data. doInsertWithSchemaEvolution(t1, @@ -874,6 +939,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") doInsert(t1, Seq((1L, "a"), (2L, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 2) doInsertWithSchemaEvolution(t1, Seq((true, "x", 2L), (false, "y", 3L)).toDF("active", "data", "id"), mode = SaveMode.Overwrite, @@ -894,6 +960,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format PARTITIONED BY (id)") doInsert(t1, Seq((1L, "a"), (2L, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 2) // Static mode overwrites the entire table. doInsertWithSchemaEvolution(t1, Seq((2L, "x", true), (3L, "y", false)).toDF("id", "data", "active"), @@ -949,8 +1016,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => sql(s"CREATE TABLE $t1 (id bigint) USING $v2Format") doInsertWithSchemaEvolution(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "b", true)).toDF("id", "data", "active")) + checkInsertMetrics(t1, numInsertedRows = 1) verifyTable(t1, Seq[(Long, String, java.lang.Boolean)]( (1L, "a", null), (2L, "b", true) @@ -965,9 +1034,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "Alice")).toDF("id", "name") .select($"id", struct(struct($"name").as("nested")).as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "Bob", 30)).toDF("id", "name", "age") .select($"id", struct(struct($"name", $"age").as("nested")).as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1L, Row(Row("Alice", null))), Row(2L, Row(Row("Bob", 30))))) @@ -981,10 +1052,12 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "Alice")).toDF("id", "name") .select($"id", struct(struct($"name").as("nested")).as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "Bob", 30)).toDF("id", "name", "age") .select($"id", struct(struct($"age", $"name").as("nested")).as("info")), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1L, Row(Row("Alice", null))), Row(2L, Row(Row("Bob", 30))))) @@ -998,9 +1071,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "Alice")).toDF("id", "name") .select($"id", array(struct($"name")).as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "Bob", 30)).toDF("id", "name", "age") .select($"id", array(struct($"name", $"age")).as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq( @@ -1016,9 +1091,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "A", "Alice")).toDF("id", "key", "name") .select($"id", map($"key", struct($"name")).as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "B", "Bob", 30)).toDF("id", "key", "name", "age") .select($"id", map($"key", struct($"name", $"age")).as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq( @@ -1034,9 +1111,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "Alice", "A")).toDF("id", "name", "value") .select($"id", map(struct($"name"), $"value").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "Bob", 30, "B")).toDF("id", "name", "age", "value") .select($"id", map(struct($"name", $"age"), $"value").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq( @@ -1050,8 +1129,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id int, data string) USING $v2Format") doInsert(t1, Seq((1, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((Long.MaxValue, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1L, "a"), Row(Long.MaxValue, "b"))) @@ -1064,8 +1145,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id int, data string) USING $v2Format") doInsert(t1, Seq((1, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq(("b", Long.MaxValue)).toDF("data", "id"), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1L, "a"), Row(Long.MaxValue, "b"))) @@ -1078,8 +1161,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id int, data string) USING $v2Format") doInsert(t1, Seq((1, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((Long.MaxValue, "b", true)).toDF("id", "data", "active")) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq( @@ -1098,9 +1183,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "Alice", 100)).toDF("id", "name", "value") .select($"id", struct($"value", $"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "Bob", Long.MaxValue)).toDF("id", "name", "value") .select($"id", struct($"value", $"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT id, info.value, info.name FROM $t1"), Seq(Row(1L, 100L, "Alice"), Row(2L, Long.MaxValue, "Bob"))) @@ -1116,10 +1203,12 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "Alice", 100)).toDF("id", "name", "value") .select($"id", struct($"value", $"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "Bob", Long.MaxValue)).toDF("id", "name", "value") .select($"id", struct($"name", $"value").as("info")), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT id, info.value, info.name FROM $t1"), Seq(Row(1L, 100L, "Alice"), Row(2L, Long.MaxValue, "Bob"))) @@ -1135,9 +1224,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, 100)).toDF("id", "value") .select($"id", array(struct($"value")).as("arr"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, Long.MaxValue)).toDF("id", "value") .select($"id", array(struct($"value")).as("arr"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT id, arr[0].value FROM $t1"), Seq(Row(1L, 100L), Row(2L, Long.MaxValue))) @@ -1154,9 +1245,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "k1", 100)).toDF("id", "key", "value") .select($"id", map($"key", struct($"value")).as("m"))) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((2L, "k2", Long.MaxValue)).toDF("id", "key", "value") .select($"id", map($"key", struct($"value")).as("m"))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT id, m['k1'].value, m['k2'].value FROM $t1"), Seq(Row(1L, 100L, null), Row(2L, null, Long.MaxValue))) @@ -1171,6 +1264,7 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id int, data string) USING $v2Format") doInsert(t1, Seq((1, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq((Long.MaxValue, "b")).toDF("id", "data"), mode = SaveMode.Overwrite) checkAnswer( @@ -1185,8 +1279,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) // Inserting an int into a long column should not narrow the schema. doInsertWithSchemaEvolution(t1, Seq((2, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1L, "a"), Row(2L, "b"))) @@ -1200,9 +1296,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id float, data string) USING $v2Format") doInsert(t1, Seq((1f, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) // Inserting a double into a float should widen the schema, inserting an int into a string // should retain the string type. doInsertWithSchemaEvolution(t1, Seq((2d, 3)).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1d, "a"), Row(2d, "3"))) @@ -1239,9 +1337,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) // Insert a null value with NullType - should not change the target column type. doInsertWithSchemaEvolution(t1, Seq(2L).toDF("id").withColumn("data", lit(null))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1L, "a"), Row(2L, null))) @@ -1254,9 +1354,11 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format") doInsert(t1, Seq((1L, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) doInsertWithSchemaEvolution(t1, Seq(2L).toDF("id").withColumn("data", lit(null)), byName = true) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1L, "a"), Row(2L, null))) @@ -1271,10 +1373,12 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => doInsert(t1, Seq((1L, "Alice", 100)).toDF("id", "name", "value") .select($"id", struct($"value", $"name").as("info"))) + checkInsertMetrics(t1, numInsertedRows = 1) // Insert with NullType for nested field - should not change the struct field type. doInsertWithSchemaEvolution(t1, Seq(2L).toDF("id") .withColumn("info", struct(lit(null).as("value"), lit("Bob").as("name")))) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT id, info.value, info.name FROM $t1"), Seq(Row(1L, 100, "Alice"), Row(2L, null, "Bob"))) @@ -1288,8 +1392,10 @@ trait InsertIntoSchemaEvolutionTests { this: InsertIntoTests => withTable(t1) { sql(s"CREATE TABLE $t1 (id int, data string) USING $v2Format") doInsert(t1, Seq((1, "a")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) // Insert without schema evolution - should cast to target type, not widen. doInsert(t1, Seq((2L, "b")).toDF("id", "data")) + checkInsertMetrics(t1, numInsertedRows = 1) checkAnswer( sql(s"SELECT * FROM $t1"), Seq(Row(1, "a"), Row(2, "b"))) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala index aaf45f0f5f7a5..b902074b547cb 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala @@ -484,6 +484,16 @@ abstract class MergeIntoTableSuiteBase extends RowLevelOperationSuiteBase Row(1, 100, "hr"), // insert Row(2, 200, "finance"), // insert Row(3, 300, "hr"))) // insert + + val mergeSummary = getMergeSummary() + assert(mergeSummary.numTargetRowsInserted === 3L) + assert(mergeSummary.numTargetRowsCopied === 0L) + assert(mergeSummary.numTargetRowsUpdated === 0L) + assert(mergeSummary.numTargetRowsDeleted === 0L) + assert(mergeSummary.numTargetRowsMatchedUpdated === 0L) + assert(mergeSummary.numTargetRowsMatchedDeleted === 0L) + assert(mergeSummary.numTargetRowsNotMatchedBySourceUpdated === 0L) + assert(mergeSummary.numTargetRowsNotMatchedBySourceDeleted === 0L) } } @@ -510,6 +520,16 @@ abstract class MergeIntoTableSuiteBase extends RowLevelOperationSuiteBase Seq( Row(2, 200, "finance"), // insert Row(3, 300, "hr"))) // insert + + val mergeSummary = getMergeSummary() + assert(mergeSummary.numTargetRowsInserted === 2L) + assert(mergeSummary.numTargetRowsCopied === 0L) + assert(mergeSummary.numTargetRowsUpdated === 0L) + assert(mergeSummary.numTargetRowsDeleted === 0L) + assert(mergeSummary.numTargetRowsMatchedUpdated === 0L) + assert(mergeSummary.numTargetRowsMatchedDeleted === 0L) + assert(mergeSummary.numTargetRowsNotMatchedBySourceUpdated === 0L) + assert(mergeSummary.numTargetRowsNotMatchedBySourceDeleted === 0L) } } @@ -539,6 +559,16 @@ abstract class MergeIntoTableSuiteBase extends RowLevelOperationSuiteBase Row(1, 100, "hr"), // insert Row(2, 200, "finance"), // insert Row(3, 300, "hr"))) // insert + + val mergeSummary = getMergeSummary() + assert(mergeSummary.numTargetRowsInserted === 3L) + assert(mergeSummary.numTargetRowsCopied === 0L) + assert(mergeSummary.numTargetRowsUpdated === 0L) + assert(mergeSummary.numTargetRowsDeleted === 0L) + assert(mergeSummary.numTargetRowsMatchedUpdated === 0L) + assert(mergeSummary.numTargetRowsMatchedDeleted === 0L) + assert(mergeSummary.numTargetRowsNotMatchedBySourceUpdated === 0L) + assert(mergeSummary.numTargetRowsNotMatchedBySourceDeleted === 0L) } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/V1WriteFallbackSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/V1WriteFallbackSuite.scala index 721b86593bacb..3e48c5222e6f2 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/connector/V1WriteFallbackSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/V1WriteFallbackSuite.scala @@ -253,6 +253,9 @@ class V1WriteFallbackSessionCatalogSuite extends InsertIntoTests(supportsDynamicOverwrite = false, includeSQLOnlyTests = true) with SessionCatalogTest[InMemoryTableWithV1Fallback, V1FallbackTableCatalog] { + // V1 fallback writes do not flow through V2TableWriteExec, so no InsertSummary is emitted. + override protected def checkInsertMetrics(tableName: String, numInsertedRows: Long): Unit = () + override protected val v2Format = classOf[InMemoryV1Provider].getName override protected val catalogClassName: String = classOf[V1FallbackTableCatalog].getName override protected val catalogAndNamespace: String = "" From 58ee7e9432e117803337ceae97b060003f08cc59 Mon Sep 17 00:00:00 2001 From: Matt Zhang <67244273+mzhang@users.noreply.github.com> Date: Thu, 14 May 2026 09:15:08 +0800 Subject: [PATCH 118/286] [SPARK-56844][SQL] Support ArrayType / MapType / StructType in ConstantColumnVector and FileSourceMetadataAttribute ### What changes were proposed in this pull request? Allow `ArrayType`, `MapType`, and `StructType` in file source constant metadata attributes. Concretely: - `FileSourceMetadataAttribute.isSupportedType` allows complex types recursively, contingent on their element types being supported. Behavior for primitives, decimal, string, binary, interval, and variant is unchanged. - `ColumnVectorUtils.populate` gains struct/array/map branches: - Struct: recurse into pre-allocated child `ConstantColumnVector`s. - Array/map: allocate a one-row `OffHeapColumnVector` backing and reuse the existing `RowToColumnConverter` (wrapped in a single-field struct schema) to write the constant value. The view is handed to the constant vector along with ownership of the backing. - `ConstantColumnVector` gains optional ownership of a backing `WritableColumnVector` (closed by `close()`), exposed via new `setArrayWithBacking` / `setMapWithBacking`. The original `setArray` / `setMap` are unchanged (caller retains ownership). - `ConstantColumnVector`'s constructor pre-allocates struct children so `populate`'s struct recursion has a target. `setChild` closes any previously-set child to avoid leaking the auto-allocated one. Notes on correctness: the recursive copy for array and map reuses `RowToColumnConverter`, which already drives row-to-columnar conversion across all supported types (`RowToColumnarExec`). No new per-type dispatch logic is introduced. ### Why are the changes needed? `FileSourceMetadataAttribute.isSupportedType` is the lone gate preventing complex-typed file source constant metadata; the underlying machinery already supports them. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New cases in `ColumnVectorUtilsSuite` replacing the previous "not supported" assertions: - `fill array of ints` - `fill array of strings` - `fill map of int -> boolean` - `fill struct` - `fill nested array` (covers element-level nulls) - `fill null array` Existing `ConstantColumnVectorSuite` cases continue to exercise the same paths. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (Anthropic) Closes #55844 from mzhang/mzhang/SPARK-56844-constant-column-vector-complex-types. Lead-authored-by: Matt Zhang <67244273+mzhang@users.noreply.github.com> Co-authored-by: Matt Zhang Signed-off-by: Wenchen Fan (cherry picked from commit e19bc35c7c7d857bb9cfa660ad8bf554dbf308c2) Signed-off-by: Wenchen Fan --- .../expressions/namedExpressions.scala | 20 +-- .../orc/OrcColumnarBatchReader.java | 3 +- .../VectorizedParquetRecordReader.java | 2 +- .../vectorized/ColumnVectorUtils.java | 45 ++++++- .../vectorized/ConstantColumnVector.java | 47 ++++++- .../execution/datasources/FileScanRDD.scala | 11 +- .../vectorized/ColumnVectorUtilsSuite.scala | 117 +++++++++++++++--- 7 files changed, 212 insertions(+), 33 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala index ccefdc0999ea8..d27f140d083b9 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala @@ -727,14 +727,20 @@ object FileSourceMetadataAttribute { * * The set of supported types is limited by [[ColumnVectorUtils.populate]], which the constant * file metadata implementation relies on. In general, types that can be partition columns are - * supported (including most primitive types). Notably unsupported types include [[ObjectType]], - * [[UserDefinedType]], and the complex types ([[StructType]], [[MapType]], [[ArrayType]]). + * supported (including most primitive types), plus the complex types [[ArrayType]], + * [[MapType]], and [[StructType]] (recursively, as long as their element types are supported). + * Notably unsupported types include [[ObjectType]] and [[UserDefinedType]]. */ - def isSupportedType(dataType: DataType): Boolean = PhysicalDataType(dataType) match { - // PhysicalPrimitiveType covers: Boolean, Byte, Double, Float, Integer, Long, Null, Short - case _: PhysicalPrimitiveType | _: PhysicalDecimalType => true - case PhysicalBinaryType | PhysicalStringType(_) | PhysicalCalendarIntervalType => true - case _ => false + def isSupportedType(dataType: DataType): Boolean = dataType match { + case ArrayType(elementType, _) => isSupportedType(elementType) + case MapType(keyType, valueType, _) => isSupportedType(keyType) && isSupportedType(valueType) + case st: StructType => st.fields.forall(f => isSupportedType(f.dataType)) + case _ => PhysicalDataType(dataType) match { + // PhysicalPrimitiveType covers: Boolean, Byte, Double, Float, Integer, Long, Null, Short + case _: PhysicalPrimitiveType | _: PhysicalDecimalType => true + case PhysicalBinaryType | PhysicalStringType(_) | PhysicalCalendarIntervalType => true + case _ => false + } } /** Returns the type unchanged if valid; otherwise throws [[IllegalArgumentException]]. */ diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java index 1c09cc9f7ff26..ea33aa1e23254 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java @@ -182,7 +182,8 @@ public void initBatch( DataType dt = requiredFields[i].dataType(); if (requestedPartitionColIds[i] != -1) { ConstantColumnVector partitionCol = new ConstantColumnVector(capacity, dt); - ColumnVectorUtils.populate(partitionCol, partitionValues, requestedPartitionColIds[i]); + ColumnVectorUtils.populate( + partitionCol, partitionValues, requestedPartitionColIds[i], memoryMode); orcVectorWrappers[i] = partitionCol; } else { int colId = requestedDataColIds[i]; diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java index 72125701fd49e..5e782433f5576 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java @@ -303,7 +303,7 @@ private void initBatch( int partitionIdx = sparkSchema.fields().length; for (int i = 0; i < partitionColumns.fields().length; i++) { ColumnVectorUtils.populate( - (ConstantColumnVector) vectors[i + partitionIdx], partitionValues, i); + (ConstantColumnVector) vectors[i + partitionIdx], partitionValues, i, MEMORY_MODE); } } diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java index 9ff385c995ff5..1ca9290e3b7c2 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java @@ -31,8 +31,10 @@ import org.apache.spark.memory.MemoryMode; import org.apache.spark.sql.Row; import org.apache.spark.sql.catalyst.InternalRow; +import org.apache.spark.sql.catalyst.expressions.GenericInternalRow; import org.apache.spark.sql.catalyst.types.*; import org.apache.spark.sql.catalyst.util.DateTimeUtils; +import org.apache.spark.sql.execution.RowToColumnConverter; import org.apache.spark.sql.types.*; import org.apache.spark.sql.vectorized.ColumnarArray; import org.apache.spark.sql.vectorized.ColumnarBatch; @@ -49,9 +51,22 @@ public class ColumnVectorUtils { /** - * Populates the value of `row[fieldIdx]` into `ConstantColumnVector`. + * Populates the value of `row[fieldIdx]` into `ConstantColumnVector`. For complex types + * (array / map) this allocates a small backing `WritableColumnVector` on-heap by default. Use + * the {@link #populate(ConstantColumnVector, InternalRow, int, MemoryMode)} overload to control + * the backing memory mode. */ public static void populate(ConstantColumnVector col, InternalRow row, int fieldIdx) { + populate(col, row, fieldIdx, MemoryMode.ON_HEAP); + } + + /** + * Populates the value of `row[fieldIdx]` into `ConstantColumnVector`. For array / map values, + * `memMode` selects on-heap vs off-heap allocation for the backing `WritableColumnVector` that + * holds the constant element data; it has no effect on primitive types. + */ + public static void populate( + ConstantColumnVector col, InternalRow row, int fieldIdx, MemoryMode memMode) { DataType t = col.dataType(); PhysicalDataType pdt = PhysicalDataType.apply(t); @@ -93,6 +108,34 @@ public static void populate(ConstantColumnVector col, InternalRow row, int field col.setCalendarInterval((CalendarInterval) row.get(fieldIdx, t)); } else if (pdt instanceof PhysicalVariantType) { col.setVariant((VariantVal)row.get(fieldIdx, t)); + } else if (pdt instanceof PhysicalStructType) { + StructType st = (StructType) t; + InternalRow inner = row.getStruct(fieldIdx, st.fields().length); + InternalRow tmpRow = new GenericInternalRow(1); + for (int i = 0; i < st.fields().length; i++) { + StructField field = st.fields()[i]; + tmpRow.update(0, inner.isNullAt(i) ? null : inner.get(i, field.dataType())); + // ConstantColumnVector's constructor pre-allocates struct children, so the recursive + // populate call below has a target vector to write into. + populate((ConstantColumnVector) col.getChild(i), tmpRow, 0, memMode); + } + } else if (pdt instanceof PhysicalArrayType || pdt instanceof PhysicalMapType) { + // Allocate a 1-row backing vector (on-heap or off-heap per `memMode`) to hold the + // constant complex value. + WritableColumnVector backing = memMode == MemoryMode.OFF_HEAP + ? new OffHeapColumnVector(1, t) + : new OnHeapColumnVector(1, t); + // Reuse RowToColumnConverter by wrapping `t` as a single-field struct schema and + // converting the one-row input. This recursively handles all element types correctly. + StructType wrapperSchema = new StructType().add("v", t, true); + RowToColumnConverter converter = new RowToColumnConverter(wrapperSchema); + InternalRow wrapped = new GenericInternalRow(new Object[]{row.get(fieldIdx, t)}); + converter.convert(wrapped, new WritableColumnVector[]{backing}); + if (pdt instanceof PhysicalArrayType) { + col.setArrayWithBacking(backing.getArray(0), backing); + } else { + col.setMapWithBacking(backing.getMap(0), backing); + } } else { throw new RuntimeException(String.format("DataType %s is not supported" + " in column vectorized reader.", t.sql())); diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java index cd2a821698853..094d6edb6d259 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java @@ -49,6 +49,8 @@ public class ConstantColumnVector extends ColumnVector { private ConstantColumnVector[] childData; private ColumnarArray arrayData; private ColumnarMap mapData; + // Optionally owned backing storage for arrayData / mapData. Closed by close(). + private WritableColumnVector ownedBacking; private final int numRows; @@ -62,6 +64,9 @@ public ConstantColumnVector(int numRows, DataType type) { if (type instanceof StructType structType) { this.childData = new ConstantColumnVector[structType.fields().length]; + for (int i = 0; i < structType.fields().length; i++) { + this.childData[i] = new ConstantColumnVector(1, structType.fields()[i].dataType()); + } } else if (type instanceof CalendarIntervalType) { // Three columns. Months as int. Days as Int. Microseconds as Long. this.childData = new ConstantColumnVector[3]; @@ -97,6 +102,10 @@ public void close() { } arrayData = null; mapData = null; + if (ownedBacking != null) { + ownedBacking.close(); + ownedBacking = null; + } } @Override @@ -218,24 +227,51 @@ public ColumnarArray getArray(int rowId) { } /** - * Sets the `ColumnarArray` `value` for all rows + * Sets the `ColumnarArray` `value` for all rows. The caller retains ownership of the backing + * storage for `value`; use `setArrayWithBacking` if this vector should own and close it. */ public void setArray(ColumnarArray value) { arrayData = value; } + /** + * Sets the `ColumnarArray` `value` for all rows and takes ownership of `backing`, which will be + * closed when this vector is closed. + */ + public void setArrayWithBacking(ColumnarArray value, WritableColumnVector backing) { + arrayData = value; + replaceOwnedBacking(backing); + } + @Override public ColumnarMap getMap(int ordinal) { return mapData; } /** - * Sets the `ColumnarMap` `value` for all rows + * Sets the `ColumnarMap` `value` for all rows. The caller retains ownership of the backing + * storage for `value`; use `setMapWithBacking` if this vector should own and close it. */ public void setMap(ColumnarMap value) { mapData = value; } + /** + * Sets the `ColumnarMap` `value` for all rows and takes ownership of `backing`, which will be + * closed when this vector is closed. + */ + public void setMapWithBacking(ColumnarMap value, WritableColumnVector backing) { + mapData = value; + replaceOwnedBacking(backing); + } + + private void replaceOwnedBacking(WritableColumnVector backing) { + if (ownedBacking != null && ownedBacking != backing) { + ownedBacking.close(); + } + ownedBacking = backing; + } + @Override public Decimal getDecimal(int rowId, int precision, int scale) { // copy and modify from WritableColumnVector @@ -303,9 +339,14 @@ public ColumnVector getChild(int ordinal) { } /** - * Sets the child `ConstantColumnVector` `value` at the given ordinal for all rows + * Sets the child `ConstantColumnVector` `value` at the given ordinal for all rows. Closes any + * previously-set child at this ordinal (e.g., one auto-allocated by the constructor) to avoid + * leaking its backing storage. */ public void setChild(int ordinal, ConstantColumnVector value) { + if (childData[ordinal] != null && childData[ordinal] != value) { + childData[ordinal].close(); + } childData[ordinal] = value; } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala index 5dc13ccee9ce0..ac1e87de863e1 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala @@ -27,6 +27,7 @@ import org.apache.hadoop.security.AccessControlException import org.apache.spark.{Partition => RDDPartition, TaskContext} import org.apache.spark.deploy.SparkHadoopUtil import org.apache.spark.internal.LogKeys.{CURRENT_FILE, PATH} +import org.apache.spark.memory.MemoryMode import org.apache.spark.paths.SparkPath import org.apache.spark.rdd.{InputFileBlockHolder, RDD} import org.apache.spark.sql.SparkSession @@ -89,6 +90,14 @@ class FileScanRDD( private val ignoreCorruptFiles = options.ignoreCorruptFiles private val ignoreMissingFiles = options.ignoreMissingFiles + // Evaluated on the driver (sparkSession is @transient) and serialized to executors so the + // `compute` iterator below can pass it through to ColumnVectorUtils.populate. + private val memoryMode: MemoryMode = + if (sparkSession.sessionState.conf.offHeapColumnVectorEnabled) { + MemoryMode.OFF_HEAP + } else { + MemoryMode.ON_HEAP + } override def compute(split: RDDPartition, context: TaskContext): Iterator[InternalRow] = { val iterator = new Iterator[Object] with AutoCloseable { @@ -183,7 +192,7 @@ class FileScanRDD( } val columnVector = new ConstantColumnVector(c.numRows(), attr.dataType) - ColumnVectorUtils.populate(columnVector, tmpRow, 0) + ColumnVectorUtils.populate(columnVector, tmpRow, 0, memoryMode) columnVector }.toArray } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorUtilsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorUtilsSuite.scala index 6205484d6be70..b1c0d6c1d7d51 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorUtilsSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorUtilsSuite.scala @@ -19,6 +19,7 @@ package org.apache.spark.sql.execution.vectorized import org.apache.spark.SparkFunSuite import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData} import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.CalendarInterval import org.apache.spark.unsafe.types.UTF8String @@ -134,30 +135,108 @@ class ColumnVectorUtilsSuite extends SparkFunSuite { } } - testConstantColumnVector("not supported: fill map", 10, + testConstantColumnVector("fill array of ints", 10, ArrayType(IntegerType)) { vector => + val arr = new GenericArrayData(Array[Any](1, 2, 3, 4, 5)) + ColumnVectorUtils.populate(vector, InternalRow(arr), 0) + (0 until 10).foreach { i => + assert(vector.getArray(i).toIntArray === Array(1, 2, 3, 4, 5)) + } + } + + testConstantColumnVector("fill array of strings", 10, ArrayType(StringType)) { vector => + val arr = new GenericArrayData(Array[Any]( + UTF8String.fromString("a"), + UTF8String.fromString("bb"), + UTF8String.fromString("ccc"))) + ColumnVectorUtils.populate(vector, InternalRow(arr), 0) + (0 until 10).foreach { i => + val a = vector.getArray(i) + assert(a.numElements() == 3) + assert(a.getUTF8String(0) == UTF8String.fromString("a")) + assert(a.getUTF8String(1) == UTF8String.fromString("bb")) + assert(a.getUTF8String(2) == UTF8String.fromString("ccc")) + } + } + + testConstantColumnVector("fill map of int -> boolean", 10, MapType(IntegerType, BooleanType)) { vector => - val message = intercept[RuntimeException] { - ColumnVectorUtils.populate(vector, InternalRow("fakeMap"), 0) - }.getMessage - assert(message == "DataType MAP is not supported in column vectorized reader.") + val keys = new GenericArrayData(Array[Any](1, 2, 3)) + val values = new GenericArrayData(Array[Any](true, false, true)) + val map = new ArrayBasedMapData(keys, values) + ColumnVectorUtils.populate(vector, InternalRow(map), 0) + (0 until 10).foreach { i => + val m = vector.getMap(i) + assert(m.numElements() == 3) + assert(m.keyArray().toIntArray === Array(1, 2, 3)) + assert(m.valueArray().toBooleanArray === Array(true, false, true)) + } } - testConstantColumnVector("not supported: fill struct", 10, + testConstantColumnVector("fill struct", 10, new StructType() .add(StructField("name", StringType)) .add(StructField("age", IntegerType))) { vector => - val message = intercept[RuntimeException] { - ColumnVectorUtils.populate(vector, InternalRow("fakeStruct"), 0) - }.getMessage - assert(message == - "DataType STRUCT is not supported in column vectorized reader.") - } - - testConstantColumnVector("not supported: fill array", 10, - ArrayType(IntegerType)) { vector => - val message = intercept[RuntimeException] { - ColumnVectorUtils.populate(vector, InternalRow("fakeArray"), 0) - }.getMessage - assert(message == "DataType ARRAY is not supported in column vectorized reader.") + val row = InternalRow(UTF8String.fromString("jack"), 27) + ColumnVectorUtils.populate(vector, InternalRow(row), 0) + (0 until 10).foreach { i => + assert(vector.getChild(0).getUTF8String(i) == UTF8String.fromString("jack")) + assert(vector.getChild(1).getInt(i) == 27) + } + } + + testConstantColumnVector("fill struct with null field", 10, + new StructType() + .add(StructField("name", StringType, nullable = true)) + .add(StructField("age", IntegerType))) { vector => + val row = InternalRow(null, 27) + ColumnVectorUtils.populate(vector, InternalRow(row), 0) + (0 until 10).foreach { i => + assert(vector.getChild(0).isNullAt(i)) + assert(vector.getChild(1).getInt(i) == 27) + } + } + + testConstantColumnVector("fill nested struct", 10, + new StructType() + .add(StructField("inner", + new StructType() + .add(StructField("k", StringType)) + .add(StructField("v", IntegerType)))) + .add(StructField("flag", BooleanType))) { vector => + val inner = InternalRow(UTF8String.fromString("a"), 1) + val outer = InternalRow(inner, true) + ColumnVectorUtils.populate(vector, InternalRow(outer), 0) + (0 until 10).foreach { i => + val s = vector.getChild(0) + assert(s.getChild(0).getUTF8String(i) == UTF8String.fromString("a")) + assert(s.getChild(1).getInt(i) == 1) + assert(vector.getChild(1).getBoolean(i)) + } + } + + testConstantColumnVector("fill nested array", 10, + ArrayType(new StructType() + .add(StructField("k", StringType)) + .add(StructField("v", IntegerType)))) { vector => + val structs = new GenericArrayData(Array[Any]( + InternalRow(UTF8String.fromString("a"), 1), + InternalRow(UTF8String.fromString("bb"), 2), + InternalRow(null, 3))) + ColumnVectorUtils.populate(vector, InternalRow(structs), 0) + (0 until 10).foreach { i => + val a = vector.getArray(i) + assert(a.numElements() == 3) + assert(a.getStruct(0, 2).getUTF8String(0) == UTF8String.fromString("a")) + assert(a.getStruct(0, 2).getInt(1) == 1) + assert(a.getStruct(1, 2).getUTF8String(0) == UTF8String.fromString("bb")) + assert(a.getStruct(1, 2).getInt(1) == 2) + assert(a.getStruct(2, 2).isNullAt(0)) + assert(a.getStruct(2, 2).getInt(1) == 3) + } + } + + testConstantColumnVector("fill null array", 10, ArrayType(IntegerType)) { vector => + ColumnVectorUtils.populate(vector, InternalRow(null), 0) + assert(vector.hasNull) } } From 2c356ac6c2d5b8a9b5f2f6a02135149939b027fe Mon Sep 17 00:00:00 2001 From: Dilip Biswal Date: Thu, 14 May 2026 09:18:20 +0800 Subject: [PATCH 119/286] [SPARK-56395][CONNECT][PYTHON] Add NEAREST BY DataFrame API Builds on the catalyst-side merged in SPARK-56395 [(link).](https://github.com/apache/spark/pull/55629) Adds the DataFrame `nearestByJoin` method in Scala / Java / PySpark and wires up Spark Connect: API completeness. The prior PR exposed `NEAREST BY` only via SQL; this PR brings the same capability to DataFrame / PySpark / Spark Connect. // Scala ``` users.nearestByJoin( products, -abs(users("score") - products("pscore")), numResults = 1, mode = "exact", direction = "similarity", joinType = "leftouter") ``` // PySpark ``` users.nearestByJoin( products, -sf.abs(users.score - products.pscore), 1, "exact", "similarity", joinType="leftouter", ).select("user_id", "product").show() ``` DataFrameNearestByJoinSuite,RewriteNearestByJoinSuite, python doctests Generated-by: Claude Code (Opus 4.7), human-reviewed and tested Closes #55682 from dilipbiswal/SPARK-56395-DF-CONNECT2. Authored-by: Dilip Biswal Signed-off-by: Wenchen Fan (cherry picked from commit 13380e780e5e398d4f498f0a97fe8b97257c80bb) Signed-off-by: Wenchen Fan --- dev/sparktestsupport/modules.py | 2 + project/MimaExcludes.scala | 2 + .../reference/pyspark.sql/dataframe.rst | 1 + python/pyspark/errors/error-conditions.json | 28 ++ python/pyspark/sql/classic/dataframe.py | 15 + python/pyspark/sql/connect/dataframe.py | 24 + python/pyspark/sql/connect/plan.py | 102 ++++ .../sql/connect/proto/relations_pb2.py | 350 +++++++------- .../sql/connect/proto/relations_pb2.pyi | 85 ++++ python/pyspark/sql/dataframe.py | 67 +++ .../connect/test_parity_nearest_by_join.py | 30 ++ .../pyspark/sql/tests/test_nearest_by_join.py | 270 +++++++++++ .../scala/org/apache/spark/sql/Dataset.scala | 70 +++ .../plans/NearestByJoinValidation.scala | 43 ++ .../spark/sql/catalyst/plans/joinTypes.scala | 16 +- .../plans/logical/NearestByJoin.scala | 6 +- .../sql/DataFrameNearestByJoinSuite.scala | 103 ++++ .../spark/sql/PlanGenerationTestSuite.scala | 23 + .../protobuf/spark/connect/relations.proto | 31 ++ .../apache/spark/sql/connect/Dataset.scala | 91 ++++ ...restByJoin_inner_approx_similarity.explain | 5 + ...estByJoin_leftouter_exact_distance.explain | 5 + ...nearestByJoin_inner_approx_similarity.json | 109 +++++ ...stByJoin_inner_approx_similarity.proto.bin | Bin 0 -> 708 bytes ...earestByJoin_leftouter_exact_distance.json | 109 +++++ ...tByJoin_leftouter_exact_distance.proto.bin | Bin 0 -> 709 bytes .../connect/planner/SparkConnectPlanner.scala | 24 + .../apache/spark/sql/classic/Dataset.scala | 60 +++ .../sql/DataFrameNearestByJoinSuite.scala | 444 ++++++++++++++++++ 29 files changed, 1930 insertions(+), 185 deletions(-) create mode 100644 python/pyspark/sql/tests/connect/test_parity_nearest_by_join.py create mode 100644 python/pyspark/sql/tests/test_nearest_by_join.py create mode 100644 sql/api/src/main/scala/org/apache/spark/sql/catalyst/plans/NearestByJoinValidation.scala create mode 100644 sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/DataFrameNearestByJoinSuite.scala create mode 100644 sql/connect/common/src/test/resources/query-tests/explain-results/nearestByJoin_inner_approx_similarity.explain create mode 100644 sql/connect/common/src/test/resources/query-tests/explain-results/nearestByJoin_leftouter_exact_distance.explain create mode 100644 sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_inner_approx_similarity.json create mode 100644 sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_inner_approx_similarity.proto.bin create mode 100644 sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_leftouter_exact_distance.json create mode 100644 sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_leftouter_exact_distance.proto.bin create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/DataFrameNearestByJoinSuite.scala diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py index 3dd001ea4cffb..41141e8bbe10d 100644 --- a/dev/sparktestsupport/modules.py +++ b/dev/sparktestsupport/modules.py @@ -611,6 +611,7 @@ def __hash__(self): "pyspark.sql.tests.test_readwriter", "pyspark.sql.tests.test_serde", "pyspark.sql.tests.test_session", + "pyspark.sql.tests.test_nearest_by_join", "pyspark.sql.tests.test_subquery", "pyspark.sql.tests.test_types", "pyspark.sql.tests.test_geographytype", @@ -1173,6 +1174,7 @@ def __hash__(self): "pyspark.sql.tests.connect.test_parity_observation", "pyspark.sql.tests.connect.test_parity_repartition", "pyspark.sql.tests.connect.test_parity_stat", + "pyspark.sql.tests.connect.test_parity_nearest_by_join", "pyspark.sql.tests.connect.test_parity_subquery", "pyspark.sql.tests.connect.test_parity_types", "pyspark.sql.tests.connect.test_parity_column", diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index bb0bc97321c34..ec9146459d8d4 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -58,6 +58,8 @@ object MimaExcludes { ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.addTaskInterruptListener"), // [SPARK-56700][SS] Make DataStreamReader.name public ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.streaming.DataStreamReader.name") + // [SPARK-56395][SQL] Add NEAREST BY top-K ranking join + ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.Dataset.nearestByJoin") ) // Exclude rules for 4.1.x from 4.0.0 diff --git a/python/docs/source/reference/pyspark.sql/dataframe.rst b/python/docs/source/reference/pyspark.sql/dataframe.rst index 9652eb7c42758..91cf0961318bf 100644 --- a/python/docs/source/reference/pyspark.sql/dataframe.rst +++ b/python/docs/source/reference/pyspark.sql/dataframe.rst @@ -84,6 +84,7 @@ DataFrame DataFrame.metadataColumn DataFrame.melt DataFrame.na + DataFrame.nearestByJoin DataFrame.observe DataFrame.offset DataFrame.orderBy diff --git a/python/pyspark/errors/error-conditions.json b/python/pyspark/errors/error-conditions.json index 485d48b6d7d5f..7cc5a73e254bd 100644 --- a/python/pyspark/errors/error-conditions.json +++ b/python/pyspark/errors/error-conditions.json @@ -602,6 +602,34 @@ "Multiple pipeline spec files found in the directory ``. Please remove one or choose a particular one with the --spec argument." ] }, + "NEAREST_BY_JOIN": { + "message": [ + "Invalid nearest-by join." + ], + "sub_class": { + "NUM_RESULTS_OUT_OF_RANGE": { + "message": [ + "The number of results must be between and . Update the literal in `APPROX NEAREST BY ...` (or `EXACT NEAREST BY ...`) to fall within that range." + ] + }, + "UNSUPPORTED_DIRECTION": { + "message": [ + "Unsupported nearest-by join direction ''. Supported nearest-by join directions include: ." + ] + }, + "UNSUPPORTED_JOIN_TYPE": { + "message": [ + "Unsupported nearest-by join type . Supported types: ." + ] + }, + "UNSUPPORTED_MODE": { + "message": [ + "Unsupported nearest-by join mode ''. Supported modes include: ." + ] + } + }, + "sqlState": "42604" + }, "NEGATIVE_VALUE": { "message": [ "Value for `` must be greater than or equal to 0, got ''." diff --git a/python/pyspark/sql/classic/dataframe.py b/python/pyspark/sql/classic/dataframe.py index 3501f01135d80..727af95485944 100644 --- a/python/pyspark/sql/classic/dataframe.py +++ b/python/pyspark/sql/classic/dataframe.py @@ -813,6 +813,21 @@ def lateralJoin( jdf = self._jdf.lateralJoin(other._jdf, on._jc, how) return DataFrame(jdf, self.sparkSession) + def nearestByJoin( + self, + other: ParentDataFrame, + rankingExpression: Column, + numResults: int, + mode: str, + direction: str, + *, + joinType: str = "inner", + ) -> ParentDataFrame: + jdf = self._jdf.nearestByJoin( + other._jdf, rankingExpression._jc, int(numResults), mode, direction, joinType + ) + return DataFrame(jdf, self.sparkSession) + # TODO(SPARK-22947): Fix the DataFrame API. def _joinAsOf( self, diff --git a/python/pyspark/sql/connect/dataframe.py b/python/pyspark/sql/connect/dataframe.py index c6602e08fac4c..b0a9692f289ad 100644 --- a/python/pyspark/sql/connect/dataframe.py +++ b/python/pyspark/sql/connect/dataframe.py @@ -726,6 +726,30 @@ def lateralJoin( session=self._session, ) + def nearestByJoin( + self, + other: ParentDataFrame, + rankingExpression: Column, + numResults: int, + mode: str, + direction: str, + *, + joinType: str = "inner", + ) -> ParentDataFrame: + other = self._check_same_session(other) + return DataFrame( + plan.NearestByJoin( + left=self._plan, + right=other._plan, + ranking_expression=rankingExpression, + num_results=int(numResults), + join_type=joinType, + mode=mode, + direction=direction, + ), + session=self._session, + ) + def _joinAsOf( self, other: ParentDataFrame, diff --git a/python/pyspark/sql/connect/plan.py b/python/pyspark/sql/connect/plan.py index 8e13cf3606570..540d81ffc6907 100644 --- a/python/pyspark/sql/connect/plan.py +++ b/python/pyspark/sql/connect/plan.py @@ -1345,6 +1345,108 @@ def _repr_html_(self) -> str: """ +# Acceptance lists for `nearestByJoin`. Must stay aligned with `NearestByJoinValidation` in +# `sql/api/.../catalyst/plans/NearestByJoinValidation.scala`. +_NEAREST_BY_JOIN_MAX_NUM_RESULTS = 100000 +_NEAREST_BY_JOIN_SUPPORTED_JOIN_TYPES = frozenset({"inner", "leftouter", "left"}) +_NEAREST_BY_JOIN_SUPPORTED_JOIN_TYPE_DISPLAY = "'INNER', 'LEFT OUTER'" +_NEAREST_BY_JOIN_SUPPORTED_MODES = ("approx", "exact") +_NEAREST_BY_JOIN_SUPPORTED_DIRECTIONS = ("distance", "similarity") + + +class NearestByJoin(LogicalPlan): + def __init__( + self, + left: Optional[LogicalPlan], + right: LogicalPlan, + ranking_expression: Column, + num_results: int, + join_type: str, + mode: str, + direction: str, + ) -> None: + super().__init__(left, self._collect_references([ranking_expression])) + self.left = cast(LogicalPlan, left) + self.right = right + self.ranking_expression = ranking_expression + # Mirror of the Scala `Dataset.validateNearestByJoinArgs` validator -- raises the same + # `NEAREST_BY_JOIN.*` error classes the server would, so the user sees a consistent + # error regardless of where the check fires. + if num_results < 1 or num_results > _NEAREST_BY_JOIN_MAX_NUM_RESULTS: + raise AnalysisException( + errorClass="NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE", + messageParameters={ + "numResults": str(num_results), + "min": "1", + "max": str(_NEAREST_BY_JOIN_MAX_NUM_RESULTS), + }, + ) + if join_type.lower().replace("_", "") not in _NEAREST_BY_JOIN_SUPPORTED_JOIN_TYPES: + raise AnalysisException( + errorClass="NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE", + messageParameters={ + "joinType": join_type, + "supported": _NEAREST_BY_JOIN_SUPPORTED_JOIN_TYPE_DISPLAY, + }, + ) + if mode.lower() not in _NEAREST_BY_JOIN_SUPPORTED_MODES: + raise AnalysisException( + errorClass="NEAREST_BY_JOIN.UNSUPPORTED_MODE", + messageParameters={ + "mode": mode, + "supported": "'" + "', '".join(_NEAREST_BY_JOIN_SUPPORTED_MODES) + "'", + }, + ) + if direction.lower() not in _NEAREST_BY_JOIN_SUPPORTED_DIRECTIONS: + raise AnalysisException( + errorClass="NEAREST_BY_JOIN.UNSUPPORTED_DIRECTION", + messageParameters={ + "direction": direction, + "supported": "'" + "', '".join(_NEAREST_BY_JOIN_SUPPORTED_DIRECTIONS) + "'", + }, + ) + self.num_results = int(num_results) + self.join_type = join_type + self.mode = mode + self.direction = direction + + def plan(self, session: "SparkConnectClient") -> proto.Relation: + plan = self._create_proto_relation() + plan.nearest_by_join.left.CopyFrom(self.left.plan(session)) + plan.nearest_by_join.right.CopyFrom(self.right.plan(session)) + plan.nearest_by_join.ranking_expression.CopyFrom(self.ranking_expression.to_plan(session)) + plan.nearest_by_join.num_results = self.num_results + plan.nearest_by_join.join_type = self.join_type + plan.nearest_by_join.mode = self.mode + plan.nearest_by_join.direction = self.direction + return self._with_relations(plan, session) + + @property + def observations(self) -> Dict[str, "Observation"]: + return {**super().observations, **self.right.observations} + + def print(self, indent: int = 0) -> str: + i = " " * indent + o = " " * (indent + LogicalPlan.INDENT) + n = indent + LogicalPlan.INDENT * 2 + return ( + f"{i}\n{o}" + f"left=\n{self.left.print(n)}\n{o}right=\n{self.right.print(n)}" + ) + + def _repr_html_(self) -> str: + return f""" +
    +
  • + NearestByJoin
    + Left: {self.left._repr_html_()} + Right: {self.right._repr_html_()} +
    • +
+ """ + + class SetOperation(LogicalPlan): def __init__( self, diff --git a/python/pyspark/sql/connect/proto/relations_pb2.py b/python/pyspark/sql/connect/proto/relations_pb2.py index d024c6a07ada8..f63b61fc344ef 100644 --- a/python/pyspark/sql/connect/proto/relations_pb2.py +++ b/python/pyspark/sql/connect/proto/relations_pb2.py @@ -44,7 +44,7 @@ DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile( - b'\n\x1dspark/connect/relations.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1fspark/connect/expressions.proto\x1a\x19spark/connect/types.proto\x1a\x1bspark/connect/catalog.proto\x1a\x1aspark/connect/common.proto\x1a\x1dspark/connect/ml_common.proto"\xd9\x1e\n\x08Relation\x12\x35\n\x06\x63ommon\x18\x01 \x01(\x0b\x32\x1d.spark.connect.RelationCommonR\x06\x63ommon\x12)\n\x04read\x18\x02 \x01(\x0b\x32\x13.spark.connect.ReadH\x00R\x04read\x12\x32\n\x07project\x18\x03 \x01(\x0b\x32\x16.spark.connect.ProjectH\x00R\x07project\x12/\n\x06\x66ilter\x18\x04 \x01(\x0b\x32\x15.spark.connect.FilterH\x00R\x06\x66ilter\x12)\n\x04join\x18\x05 \x01(\x0b\x32\x13.spark.connect.JoinH\x00R\x04join\x12\x34\n\x06set_op\x18\x06 \x01(\x0b\x32\x1b.spark.connect.SetOperationH\x00R\x05setOp\x12)\n\x04sort\x18\x07 \x01(\x0b\x32\x13.spark.connect.SortH\x00R\x04sort\x12,\n\x05limit\x18\x08 \x01(\x0b\x32\x14.spark.connect.LimitH\x00R\x05limit\x12\x38\n\taggregate\x18\t \x01(\x0b\x32\x18.spark.connect.AggregateH\x00R\taggregate\x12&\n\x03sql\x18\n \x01(\x0b\x32\x12.spark.connect.SQLH\x00R\x03sql\x12\x45\n\x0elocal_relation\x18\x0b \x01(\x0b\x32\x1c.spark.connect.LocalRelationH\x00R\rlocalRelation\x12/\n\x06sample\x18\x0c \x01(\x0b\x32\x15.spark.connect.SampleH\x00R\x06sample\x12/\n\x06offset\x18\r \x01(\x0b\x32\x15.spark.connect.OffsetH\x00R\x06offset\x12>\n\x0b\x64\x65\x64uplicate\x18\x0e \x01(\x0b\x32\x1a.spark.connect.DeduplicateH\x00R\x0b\x64\x65\x64uplicate\x12,\n\x05range\x18\x0f \x01(\x0b\x32\x14.spark.connect.RangeH\x00R\x05range\x12\x45\n\x0esubquery_alias\x18\x10 \x01(\x0b\x32\x1c.spark.connect.SubqueryAliasH\x00R\rsubqueryAlias\x12>\n\x0brepartition\x18\x11 \x01(\x0b\x32\x1a.spark.connect.RepartitionH\x00R\x0brepartition\x12*\n\x05to_df\x18\x12 \x01(\x0b\x32\x13.spark.connect.ToDFH\x00R\x04toDf\x12U\n\x14with_columns_renamed\x18\x13 \x01(\x0b\x32!.spark.connect.WithColumnsRenamedH\x00R\x12withColumnsRenamed\x12<\n\x0bshow_string\x18\x14 \x01(\x0b\x32\x19.spark.connect.ShowStringH\x00R\nshowString\x12)\n\x04\x64rop\x18\x15 \x01(\x0b\x32\x13.spark.connect.DropH\x00R\x04\x64rop\x12)\n\x04tail\x18\x16 \x01(\x0b\x32\x13.spark.connect.TailH\x00R\x04tail\x12?\n\x0cwith_columns\x18\x17 \x01(\x0b\x32\x1a.spark.connect.WithColumnsH\x00R\x0bwithColumns\x12)\n\x04hint\x18\x18 \x01(\x0b\x32\x13.spark.connect.HintH\x00R\x04hint\x12\x32\n\x07unpivot\x18\x19 \x01(\x0b\x32\x16.spark.connect.UnpivotH\x00R\x07unpivot\x12\x36\n\tto_schema\x18\x1a \x01(\x0b\x32\x17.spark.connect.ToSchemaH\x00R\x08toSchema\x12\x64\n\x19repartition_by_expression\x18\x1b \x01(\x0b\x32&.spark.connect.RepartitionByExpressionH\x00R\x17repartitionByExpression\x12\x45\n\x0emap_partitions\x18\x1c \x01(\x0b\x32\x1c.spark.connect.MapPartitionsH\x00R\rmapPartitions\x12H\n\x0f\x63ollect_metrics\x18\x1d \x01(\x0b\x32\x1d.spark.connect.CollectMetricsH\x00R\x0e\x63ollectMetrics\x12,\n\x05parse\x18\x1e \x01(\x0b\x32\x14.spark.connect.ParseH\x00R\x05parse\x12\x36\n\tgroup_map\x18\x1f \x01(\x0b\x32\x17.spark.connect.GroupMapH\x00R\x08groupMap\x12=\n\x0c\x63o_group_map\x18 \x01(\x0b\x32\x19.spark.connect.CoGroupMapH\x00R\ncoGroupMap\x12\x45\n\x0ewith_watermark\x18! \x01(\x0b\x32\x1c.spark.connect.WithWatermarkH\x00R\rwithWatermark\x12\x63\n\x1a\x61pply_in_pandas_with_state\x18" \x01(\x0b\x32%.spark.connect.ApplyInPandasWithStateH\x00R\x16\x61pplyInPandasWithState\x12<\n\x0bhtml_string\x18# \x01(\x0b\x32\x19.spark.connect.HtmlStringH\x00R\nhtmlString\x12X\n\x15\x63\x61\x63hed_local_relation\x18$ \x01(\x0b\x32".spark.connect.CachedLocalRelationH\x00R\x13\x63\x61\x63hedLocalRelation\x12[\n\x16\x63\x61\x63hed_remote_relation\x18% \x01(\x0b\x32#.spark.connect.CachedRemoteRelationH\x00R\x14\x63\x61\x63hedRemoteRelation\x12\x8e\x01\n)common_inline_user_defined_table_function\x18& \x01(\x0b\x32\x33.spark.connect.CommonInlineUserDefinedTableFunctionH\x00R$commonInlineUserDefinedTableFunction\x12\x37\n\nas_of_join\x18\' \x01(\x0b\x32\x17.spark.connect.AsOfJoinH\x00R\x08\x61sOfJoin\x12\x85\x01\n&common_inline_user_defined_data_source\x18( \x01(\x0b\x32\x30.spark.connect.CommonInlineUserDefinedDataSourceH\x00R!commonInlineUserDefinedDataSource\x12\x45\n\x0ewith_relations\x18) \x01(\x0b\x32\x1c.spark.connect.WithRelationsH\x00R\rwithRelations\x12\x38\n\ttranspose\x18* \x01(\x0b\x32\x18.spark.connect.TransposeH\x00R\ttranspose\x12w\n unresolved_table_valued_function\x18+ \x01(\x0b\x32,.spark.connect.UnresolvedTableValuedFunctionH\x00R\x1dunresolvedTableValuedFunction\x12?\n\x0clateral_join\x18, \x01(\x0b\x32\x1a.spark.connect.LateralJoinH\x00R\x0blateralJoin\x12n\n\x1d\x63hunked_cached_local_relation\x18- \x01(\x0b\x32).spark.connect.ChunkedCachedLocalRelationH\x00R\x1a\x63hunkedCachedLocalRelation\x12K\n\x10relation_changes\x18. \x01(\x0b\x32\x1e.spark.connect.RelationChangesH\x00R\x0frelationChanges\x12\x30\n\x07\x66ill_na\x18Z \x01(\x0b\x32\x15.spark.connect.NAFillH\x00R\x06\x66illNa\x12\x30\n\x07\x64rop_na\x18[ \x01(\x0b\x32\x15.spark.connect.NADropH\x00R\x06\x64ropNa\x12\x34\n\x07replace\x18\\ \x01(\x0b\x32\x18.spark.connect.NAReplaceH\x00R\x07replace\x12\x36\n\x07summary\x18\x64 \x01(\x0b\x32\x1a.spark.connect.StatSummaryH\x00R\x07summary\x12\x39\n\x08\x63rosstab\x18\x65 \x01(\x0b\x32\x1b.spark.connect.StatCrosstabH\x00R\x08\x63rosstab\x12\x39\n\x08\x64\x65scribe\x18\x66 \x01(\x0b\x32\x1b.spark.connect.StatDescribeH\x00R\x08\x64\x65scribe\x12*\n\x03\x63ov\x18g \x01(\x0b\x32\x16.spark.connect.StatCovH\x00R\x03\x63ov\x12-\n\x04\x63orr\x18h \x01(\x0b\x32\x17.spark.connect.StatCorrH\x00R\x04\x63orr\x12L\n\x0f\x61pprox_quantile\x18i \x01(\x0b\x32!.spark.connect.StatApproxQuantileH\x00R\x0e\x61pproxQuantile\x12=\n\nfreq_items\x18j \x01(\x0b\x32\x1c.spark.connect.StatFreqItemsH\x00R\tfreqItems\x12:\n\tsample_by\x18k \x01(\x0b\x32\x1b.spark.connect.StatSampleByH\x00R\x08sampleBy\x12\x33\n\x07\x63\x61talog\x18\xc8\x01 \x01(\x0b\x32\x16.spark.connect.CatalogH\x00R\x07\x63\x61talog\x12=\n\x0bml_relation\x18\xac\x02 \x01(\x0b\x32\x19.spark.connect.MlRelationH\x00R\nmlRelation\x12\x35\n\textension\x18\xe6\x07 \x01(\x0b\x32\x14.google.protobuf.AnyH\x00R\textension\x12\x33\n\x07unknown\x18\xe7\x07 \x01(\x0b\x32\x16.spark.connect.UnknownH\x00R\x07unknownB\n\n\x08rel_type"\xe4\x03\n\nMlRelation\x12\x43\n\ttransform\x18\x01 \x01(\x0b\x32#.spark.connect.MlRelation.TransformH\x00R\ttransform\x12,\n\x05\x66\x65tch\x18\x02 \x01(\x0b\x32\x14.spark.connect.FetchH\x00R\x05\x66\x65tch\x12P\n\x15model_summary_dataset\x18\x03 \x01(\x0b\x32\x17.spark.connect.RelationH\x01R\x13modelSummaryDataset\x88\x01\x01\x1a\xeb\x01\n\tTransform\x12\x33\n\x07obj_ref\x18\x01 \x01(\x0b\x32\x18.spark.connect.ObjectRefH\x00R\x06objRef\x12=\n\x0btransformer\x18\x02 \x01(\x0b\x32\x19.spark.connect.MlOperatorH\x00R\x0btransformer\x12-\n\x05input\x18\x03 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12/\n\x06params\x18\x04 \x01(\x0b\x32\x17.spark.connect.MlParamsR\x06paramsB\n\n\x08operatorB\t\n\x07ml_typeB\x18\n\x16_model_summary_dataset"\xcb\x02\n\x05\x46\x65tch\x12\x31\n\x07obj_ref\x18\x01 \x01(\x0b\x32\x18.spark.connect.ObjectRefR\x06objRef\x12\x35\n\x07methods\x18\x02 \x03(\x0b\x32\x1b.spark.connect.Fetch.MethodR\x07methods\x1a\xd7\x01\n\x06Method\x12\x16\n\x06method\x18\x01 \x01(\tR\x06method\x12\x34\n\x04\x61rgs\x18\x02 \x03(\x0b\x32 .spark.connect.Fetch.Method.ArgsR\x04\x61rgs\x1a\x7f\n\x04\x41rgs\x12\x39\n\x05param\x18\x01 \x01(\x0b\x32!.spark.connect.Expression.LiteralH\x00R\x05param\x12/\n\x05input\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationH\x00R\x05inputB\x0b\n\targs_type"\t\n\x07Unknown"\x8e\x01\n\x0eRelationCommon\x12#\n\x0bsource_info\x18\x01 \x01(\tB\x02\x18\x01R\nsourceInfo\x12\x1c\n\x07plan_id\x18\x02 \x01(\x03H\x00R\x06planId\x88\x01\x01\x12-\n\x06origin\x18\x03 \x01(\x0b\x32\x15.spark.connect.OriginR\x06originB\n\n\x08_plan_id"\xde\x03\n\x03SQL\x12\x14\n\x05query\x18\x01 \x01(\tR\x05query\x12\x34\n\x04\x61rgs\x18\x02 \x03(\x0b\x32\x1c.spark.connect.SQL.ArgsEntryB\x02\x18\x01R\x04\x61rgs\x12@\n\x08pos_args\x18\x03 \x03(\x0b\x32!.spark.connect.Expression.LiteralB\x02\x18\x01R\x07posArgs\x12O\n\x0fnamed_arguments\x18\x04 \x03(\x0b\x32&.spark.connect.SQL.NamedArgumentsEntryR\x0enamedArguments\x12>\n\rpos_arguments\x18\x05 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0cposArguments\x1aZ\n\tArgsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x37\n\x05value\x18\x02 \x01(\x0b\x32!.spark.connect.Expression.LiteralR\x05value:\x02\x38\x01\x1a\\\n\x13NamedArgumentsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12/\n\x05value\x18\x02 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x05value:\x02\x38\x01"u\n\rWithRelations\x12+\n\x04root\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04root\x12\x37\n\nreferences\x18\x02 \x03(\x0b\x32\x17.spark.connect.RelationR\nreferences"\xcd\x05\n\x04Read\x12\x41\n\x0bnamed_table\x18\x01 \x01(\x0b\x32\x1e.spark.connect.Read.NamedTableH\x00R\nnamedTable\x12\x41\n\x0b\x64\x61ta_source\x18\x02 \x01(\x0b\x32\x1e.spark.connect.Read.DataSourceH\x00R\ndataSource\x12!\n\x0cis_streaming\x18\x03 \x01(\x08R\x0bisStreaming\x1a\xc0\x01\n\nNamedTable\x12/\n\x13unparsed_identifier\x18\x01 \x01(\tR\x12unparsedIdentifier\x12\x45\n\x07options\x18\x02 \x03(\x0b\x32+.spark.connect.Read.NamedTable.OptionsEntryR\x07options\x1a:\n\x0cOptionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1a\xcb\x02\n\nDataSource\x12\x1b\n\x06\x66ormat\x18\x01 \x01(\tH\x00R\x06\x66ormat\x88\x01\x01\x12\x1b\n\x06schema\x18\x02 \x01(\tH\x01R\x06schema\x88\x01\x01\x12\x45\n\x07options\x18\x03 \x03(\x0b\x32+.spark.connect.Read.DataSource.OptionsEntryR\x07options\x12\x14\n\x05paths\x18\x04 \x03(\tR\x05paths\x12\x1e\n\npredicates\x18\x05 \x03(\tR\npredicates\x12$\n\x0bsource_name\x18\x06 \x01(\tH\x02R\nsourceName\x88\x01\x01\x1a:\n\x0cOptionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x42\t\n\x07_formatB\t\n\x07_schemaB\x0e\n\x0c_source_nameB\x0b\n\tread_type"\xe8\x01\n\x0fRelationChanges\x12/\n\x13unparsed_identifier\x18\x01 \x01(\tR\x12unparsedIdentifier\x12\x45\n\x07options\x18\x02 \x03(\x0b\x32+.spark.connect.RelationChanges.OptionsEntryR\x07options\x12!\n\x0cis_streaming\x18\x03 \x01(\x08R\x0bisStreaming\x1a:\n\x0cOptionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01"u\n\x07Project\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12;\n\x0b\x65xpressions\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0b\x65xpressions"p\n\x06\x46ilter\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x37\n\tcondition\x18\x02 \x01(\x0b\x32\x19.spark.connect.ExpressionR\tcondition"\x95\x05\n\x04Join\x12+\n\x04left\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04left\x12-\n\x05right\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\x05right\x12@\n\x0ejoin_condition\x18\x03 \x01(\x0b\x32\x19.spark.connect.ExpressionR\rjoinCondition\x12\x39\n\tjoin_type\x18\x04 \x01(\x0e\x32\x1c.spark.connect.Join.JoinTypeR\x08joinType\x12#\n\rusing_columns\x18\x05 \x03(\tR\x0cusingColumns\x12K\n\x0ejoin_data_type\x18\x06 \x01(\x0b\x32 .spark.connect.Join.JoinDataTypeH\x00R\x0cjoinDataType\x88\x01\x01\x1a\\\n\x0cJoinDataType\x12$\n\x0eis_left_struct\x18\x01 \x01(\x08R\x0cisLeftStruct\x12&\n\x0fis_right_struct\x18\x02 \x01(\x08R\risRightStruct"\xd0\x01\n\x08JoinType\x12\x19\n\x15JOIN_TYPE_UNSPECIFIED\x10\x00\x12\x13\n\x0fJOIN_TYPE_INNER\x10\x01\x12\x18\n\x14JOIN_TYPE_FULL_OUTER\x10\x02\x12\x18\n\x14JOIN_TYPE_LEFT_OUTER\x10\x03\x12\x19\n\x15JOIN_TYPE_RIGHT_OUTER\x10\x04\x12\x17\n\x13JOIN_TYPE_LEFT_ANTI\x10\x05\x12\x17\n\x13JOIN_TYPE_LEFT_SEMI\x10\x06\x12\x13\n\x0fJOIN_TYPE_CROSS\x10\x07\x42\x11\n\x0f_join_data_type"\xdf\x03\n\x0cSetOperation\x12\x36\n\nleft_input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\tleftInput\x12\x38\n\x0bright_input\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\nrightInput\x12\x45\n\x0bset_op_type\x18\x03 \x01(\x0e\x32%.spark.connect.SetOperation.SetOpTypeR\tsetOpType\x12\x1a\n\x06is_all\x18\x04 \x01(\x08H\x00R\x05isAll\x88\x01\x01\x12\x1c\n\x07\x62y_name\x18\x05 \x01(\x08H\x01R\x06\x62yName\x88\x01\x01\x12\x37\n\x15\x61llow_missing_columns\x18\x06 \x01(\x08H\x02R\x13\x61llowMissingColumns\x88\x01\x01"r\n\tSetOpType\x12\x1b\n\x17SET_OP_TYPE_UNSPECIFIED\x10\x00\x12\x19\n\x15SET_OP_TYPE_INTERSECT\x10\x01\x12\x15\n\x11SET_OP_TYPE_UNION\x10\x02\x12\x16\n\x12SET_OP_TYPE_EXCEPT\x10\x03\x42\t\n\x07_is_allB\n\n\x08_by_nameB\x18\n\x16_allow_missing_columns"L\n\x05Limit\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x14\n\x05limit\x18\x02 \x01(\x05R\x05limit"O\n\x06Offset\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x16\n\x06offset\x18\x02 \x01(\x05R\x06offset"K\n\x04Tail\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x14\n\x05limit\x18\x02 \x01(\x05R\x05limit"\xfe\x05\n\tAggregate\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x41\n\ngroup_type\x18\x02 \x01(\x0e\x32".spark.connect.Aggregate.GroupTypeR\tgroupType\x12L\n\x14grouping_expressions\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x13groupingExpressions\x12N\n\x15\x61ggregate_expressions\x18\x04 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x14\x61ggregateExpressions\x12\x34\n\x05pivot\x18\x05 \x01(\x0b\x32\x1e.spark.connect.Aggregate.PivotR\x05pivot\x12J\n\rgrouping_sets\x18\x06 \x03(\x0b\x32%.spark.connect.Aggregate.GroupingSetsR\x0cgroupingSets\x1ao\n\x05Pivot\x12+\n\x03\x63ol\x18\x01 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x03\x63ol\x12\x39\n\x06values\x18\x02 \x03(\x0b\x32!.spark.connect.Expression.LiteralR\x06values\x1aL\n\x0cGroupingSets\x12<\n\x0cgrouping_set\x18\x01 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0bgroupingSet"\x9f\x01\n\tGroupType\x12\x1a\n\x16GROUP_TYPE_UNSPECIFIED\x10\x00\x12\x16\n\x12GROUP_TYPE_GROUPBY\x10\x01\x12\x15\n\x11GROUP_TYPE_ROLLUP\x10\x02\x12\x13\n\x0fGROUP_TYPE_CUBE\x10\x03\x12\x14\n\x10GROUP_TYPE_PIVOT\x10\x04\x12\x1c\n\x18GROUP_TYPE_GROUPING_SETS\x10\x05"\xa0\x01\n\x04Sort\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x39\n\x05order\x18\x02 \x03(\x0b\x32#.spark.connect.Expression.SortOrderR\x05order\x12 \n\tis_global\x18\x03 \x01(\x08H\x00R\x08isGlobal\x88\x01\x01\x42\x0c\n\n_is_global"\x8d\x01\n\x04\x44rop\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x33\n\x07\x63olumns\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x07\x63olumns\x12!\n\x0c\x63olumn_names\x18\x03 \x03(\tR\x0b\x63olumnNames"\xf0\x01\n\x0b\x44\x65\x64uplicate\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12!\n\x0c\x63olumn_names\x18\x02 \x03(\tR\x0b\x63olumnNames\x12\x32\n\x13\x61ll_columns_as_keys\x18\x03 \x01(\x08H\x00R\x10\x61llColumnsAsKeys\x88\x01\x01\x12.\n\x10within_watermark\x18\x04 \x01(\x08H\x01R\x0fwithinWatermark\x88\x01\x01\x42\x16\n\x14_all_columns_as_keysB\x13\n\x11_within_watermark"Y\n\rLocalRelation\x12\x17\n\x04\x64\x61ta\x18\x01 \x01(\x0cH\x00R\x04\x64\x61ta\x88\x01\x01\x12\x1b\n\x06schema\x18\x02 \x01(\tH\x01R\x06schema\x88\x01\x01\x42\x07\n\x05_dataB\t\n\x07_schema"H\n\x13\x43\x61\x63hedLocalRelation\x12\x12\n\x04hash\x18\x03 \x01(\tR\x04hashJ\x04\x08\x01\x10\x02J\x04\x08\x02\x10\x03R\x06userIdR\tsessionId"p\n\x1a\x43hunkedCachedLocalRelation\x12\x1e\n\ndataHashes\x18\x01 \x03(\tR\ndataHashes\x12#\n\nschemaHash\x18\x02 \x01(\tH\x00R\nschemaHash\x88\x01\x01\x42\r\n\x0b_schemaHash"7\n\x14\x43\x61\x63hedRemoteRelation\x12\x1f\n\x0brelation_id\x18\x01 \x01(\tR\nrelationId"\x91\x02\n\x06Sample\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x1f\n\x0blower_bound\x18\x02 \x01(\x01R\nlowerBound\x12\x1f\n\x0bupper_bound\x18\x03 \x01(\x01R\nupperBound\x12.\n\x10with_replacement\x18\x04 \x01(\x08H\x00R\x0fwithReplacement\x88\x01\x01\x12\x17\n\x04seed\x18\x05 \x01(\x03H\x01R\x04seed\x88\x01\x01\x12/\n\x13\x64\x65terministic_order\x18\x06 \x01(\x08R\x12\x64\x65terministicOrderB\x13\n\x11_with_replacementB\x07\n\x05_seed"\x91\x01\n\x05Range\x12\x19\n\x05start\x18\x01 \x01(\x03H\x00R\x05start\x88\x01\x01\x12\x10\n\x03\x65nd\x18\x02 \x01(\x03R\x03\x65nd\x12\x12\n\x04step\x18\x03 \x01(\x03R\x04step\x12*\n\x0enum_partitions\x18\x04 \x01(\x05H\x01R\rnumPartitions\x88\x01\x01\x42\x08\n\x06_startB\x11\n\x0f_num_partitions"r\n\rSubqueryAlias\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x14\n\x05\x61lias\x18\x02 \x01(\tR\x05\x61lias\x12\x1c\n\tqualifier\x18\x03 \x03(\tR\tqualifier"\x8e\x01\n\x0bRepartition\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12%\n\x0enum_partitions\x18\x02 \x01(\x05R\rnumPartitions\x12\x1d\n\x07shuffle\x18\x03 \x01(\x08H\x00R\x07shuffle\x88\x01\x01\x42\n\n\x08_shuffle"\x8e\x01\n\nShowString\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x19\n\x08num_rows\x18\x02 \x01(\x05R\x07numRows\x12\x1a\n\x08truncate\x18\x03 \x01(\x05R\x08truncate\x12\x1a\n\x08vertical\x18\x04 \x01(\x08R\x08vertical"r\n\nHtmlString\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x19\n\x08num_rows\x18\x02 \x01(\x05R\x07numRows\x12\x1a\n\x08truncate\x18\x03 \x01(\x05R\x08truncate"\\\n\x0bStatSummary\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x1e\n\nstatistics\x18\x02 \x03(\tR\nstatistics"Q\n\x0cStatDescribe\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols"e\n\x0cStatCrosstab\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ol1\x18\x02 \x01(\tR\x04\x63ol1\x12\x12\n\x04\x63ol2\x18\x03 \x01(\tR\x04\x63ol2"`\n\x07StatCov\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ol1\x18\x02 \x01(\tR\x04\x63ol1\x12\x12\n\x04\x63ol2\x18\x03 \x01(\tR\x04\x63ol2"\x89\x01\n\x08StatCorr\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ol1\x18\x02 \x01(\tR\x04\x63ol1\x12\x12\n\x04\x63ol2\x18\x03 \x01(\tR\x04\x63ol2\x12\x1b\n\x06method\x18\x04 \x01(\tH\x00R\x06method\x88\x01\x01\x42\t\n\x07_method"\xa4\x01\n\x12StatApproxQuantile\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12$\n\rprobabilities\x18\x03 \x03(\x01R\rprobabilities\x12%\n\x0erelative_error\x18\x04 \x01(\x01R\rrelativeError"}\n\rStatFreqItems\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12\x1d\n\x07support\x18\x03 \x01(\x01H\x00R\x07support\x88\x01\x01\x42\n\n\x08_support"\xb5\x02\n\x0cStatSampleBy\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12+\n\x03\x63ol\x18\x02 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x03\x63ol\x12\x42\n\tfractions\x18\x03 \x03(\x0b\x32$.spark.connect.StatSampleBy.FractionR\tfractions\x12\x17\n\x04seed\x18\x05 \x01(\x03H\x00R\x04seed\x88\x01\x01\x1a\x63\n\x08\x46raction\x12;\n\x07stratum\x18\x01 \x01(\x0b\x32!.spark.connect.Expression.LiteralR\x07stratum\x12\x1a\n\x08\x66raction\x18\x02 \x01(\x01R\x08\x66ractionB\x07\n\x05_seed"\x86\x01\n\x06NAFill\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12\x39\n\x06values\x18\x03 \x03(\x0b\x32!.spark.connect.Expression.LiteralR\x06values"\x86\x01\n\x06NADrop\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12\'\n\rmin_non_nulls\x18\x03 \x01(\x05H\x00R\x0bminNonNulls\x88\x01\x01\x42\x10\n\x0e_min_non_nulls"\xa8\x02\n\tNAReplace\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12H\n\x0creplacements\x18\x03 \x03(\x0b\x32$.spark.connect.NAReplace.ReplacementR\x0creplacements\x1a\x8d\x01\n\x0bReplacement\x12>\n\told_value\x18\x01 \x01(\x0b\x32!.spark.connect.Expression.LiteralR\x08oldValue\x12>\n\tnew_value\x18\x02 \x01(\x0b\x32!.spark.connect.Expression.LiteralR\x08newValue"X\n\x04ToDF\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12!\n\x0c\x63olumn_names\x18\x02 \x03(\tR\x0b\x63olumnNames"\xfe\x02\n\x12WithColumnsRenamed\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12i\n\x12rename_columns_map\x18\x02 \x03(\x0b\x32\x37.spark.connect.WithColumnsRenamed.RenameColumnsMapEntryB\x02\x18\x01R\x10renameColumnsMap\x12\x42\n\x07renames\x18\x03 \x03(\x0b\x32(.spark.connect.WithColumnsRenamed.RenameR\x07renames\x1a\x43\n\x15RenameColumnsMapEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1a\x45\n\x06Rename\x12\x19\n\x08\x63ol_name\x18\x01 \x01(\tR\x07\x63olName\x12 \n\x0cnew_col_name\x18\x02 \x01(\tR\nnewColName"w\n\x0bWithColumns\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x39\n\x07\x61liases\x18\x02 \x03(\x0b\x32\x1f.spark.connect.Expression.AliasR\x07\x61liases"\x86\x01\n\rWithWatermark\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x1d\n\nevent_time\x18\x02 \x01(\tR\teventTime\x12\'\n\x0f\x64\x65lay_threshold\x18\x03 \x01(\tR\x0e\x64\x65layThreshold"\x84\x01\n\x04Hint\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04name\x18\x02 \x01(\tR\x04name\x12\x39\n\nparameters\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\nparameters"\xc7\x02\n\x07Unpivot\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12+\n\x03ids\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x03ids\x12:\n\x06values\x18\x03 \x01(\x0b\x32\x1d.spark.connect.Unpivot.ValuesH\x00R\x06values\x88\x01\x01\x12\x30\n\x14variable_column_name\x18\x04 \x01(\tR\x12variableColumnName\x12*\n\x11value_column_name\x18\x05 \x01(\tR\x0fvalueColumnName\x1a;\n\x06Values\x12\x31\n\x06values\x18\x01 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x06valuesB\t\n\x07_values"z\n\tTranspose\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12>\n\rindex_columns\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0cindexColumns"}\n\x1dUnresolvedTableValuedFunction\x12#\n\rfunction_name\x18\x01 \x01(\tR\x0c\x66unctionName\x12\x37\n\targuments\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\targuments"j\n\x08ToSchema\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12/\n\x06schema\x18\x02 \x01(\x0b\x32\x17.spark.connect.DataTypeR\x06schema"\xcb\x01\n\x17RepartitionByExpression\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x42\n\x0fpartition_exprs\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0epartitionExprs\x12*\n\x0enum_partitions\x18\x03 \x01(\x05H\x00R\rnumPartitions\x88\x01\x01\x42\x11\n\x0f_num_partitions"\xe8\x01\n\rMapPartitions\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x42\n\x04\x66unc\x18\x02 \x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionR\x04\x66unc\x12"\n\nis_barrier\x18\x03 \x01(\x08H\x00R\tisBarrier\x88\x01\x01\x12"\n\nprofile_id\x18\x04 \x01(\x05H\x01R\tprofileId\x88\x01\x01\x42\r\n\x0b_is_barrierB\r\n\x0b_profile_id"\xd2\x06\n\x08GroupMap\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12L\n\x14grouping_expressions\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x13groupingExpressions\x12\x42\n\x04\x66unc\x18\x03 \x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionR\x04\x66unc\x12J\n\x13sorting_expressions\x18\x04 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x12sortingExpressions\x12<\n\rinitial_input\x18\x05 \x01(\x0b\x32\x17.spark.connect.RelationR\x0cinitialInput\x12[\n\x1cinitial_grouping_expressions\x18\x06 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x1ainitialGroupingExpressions\x12;\n\x18is_map_groups_with_state\x18\x07 \x01(\x08H\x00R\x14isMapGroupsWithState\x88\x01\x01\x12$\n\x0boutput_mode\x18\x08 \x01(\tH\x01R\noutputMode\x88\x01\x01\x12&\n\x0ctimeout_conf\x18\t \x01(\tH\x02R\x0btimeoutConf\x88\x01\x01\x12?\n\x0cstate_schema\x18\n \x01(\x0b\x32\x17.spark.connect.DataTypeH\x03R\x0bstateSchema\x88\x01\x01\x12\x65\n\x19transform_with_state_info\x18\x0b \x01(\x0b\x32%.spark.connect.TransformWithStateInfoH\x04R\x16transformWithStateInfo\x88\x01\x01\x42\x1b\n\x19_is_map_groups_with_stateB\x0e\n\x0c_output_modeB\x0f\n\r_timeout_confB\x0f\n\r_state_schemaB\x1c\n\x1a_transform_with_state_info"\xdf\x01\n\x16TransformWithStateInfo\x12\x1b\n\ttime_mode\x18\x01 \x01(\tR\x08timeMode\x12\x38\n\x16\x65vent_time_column_name\x18\x02 \x01(\tH\x00R\x13\x65ventTimeColumnName\x88\x01\x01\x12\x41\n\routput_schema\x18\x03 \x01(\x0b\x32\x17.spark.connect.DataTypeH\x01R\x0coutputSchema\x88\x01\x01\x42\x19\n\x17_event_time_column_nameB\x10\n\x0e_output_schema"\x8e\x04\n\nCoGroupMap\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12W\n\x1ainput_grouping_expressions\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x18inputGroupingExpressions\x12-\n\x05other\x18\x03 \x01(\x0b\x32\x17.spark.connect.RelationR\x05other\x12W\n\x1aother_grouping_expressions\x18\x04 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x18otherGroupingExpressions\x12\x42\n\x04\x66unc\x18\x05 \x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionR\x04\x66unc\x12U\n\x19input_sorting_expressions\x18\x06 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x17inputSortingExpressions\x12U\n\x19other_sorting_expressions\x18\x07 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x17otherSortingExpressions"\xe5\x02\n\x16\x41pplyInPandasWithState\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12L\n\x14grouping_expressions\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x13groupingExpressions\x12\x42\n\x04\x66unc\x18\x03 \x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionR\x04\x66unc\x12#\n\routput_schema\x18\x04 \x01(\tR\x0coutputSchema\x12!\n\x0cstate_schema\x18\x05 \x01(\tR\x0bstateSchema\x12\x1f\n\x0boutput_mode\x18\x06 \x01(\tR\noutputMode\x12!\n\x0ctimeout_conf\x18\x07 \x01(\tR\x0btimeoutConf"\xf4\x01\n$CommonInlineUserDefinedTableFunction\x12#\n\rfunction_name\x18\x01 \x01(\tR\x0c\x66unctionName\x12$\n\rdeterministic\x18\x02 \x01(\x08R\rdeterministic\x12\x37\n\targuments\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\targuments\x12<\n\x0bpython_udtf\x18\x04 \x01(\x0b\x32\x19.spark.connect.PythonUDTFH\x00R\npythonUdtfB\n\n\x08\x66unction"\xb1\x01\n\nPythonUDTF\x12=\n\x0breturn_type\x18\x01 \x01(\x0b\x32\x17.spark.connect.DataTypeH\x00R\nreturnType\x88\x01\x01\x12\x1b\n\teval_type\x18\x02 \x01(\x05R\x08\x65valType\x12\x18\n\x07\x63ommand\x18\x03 \x01(\x0cR\x07\x63ommand\x12\x1d\n\npython_ver\x18\x04 \x01(\tR\tpythonVerB\x0e\n\x0c_return_type"\x97\x01\n!CommonInlineUserDefinedDataSource\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12O\n\x12python_data_source\x18\x02 \x01(\x0b\x32\x1f.spark.connect.PythonDataSourceH\x00R\x10pythonDataSourceB\r\n\x0b\x64\x61ta_source"K\n\x10PythonDataSource\x12\x18\n\x07\x63ommand\x18\x01 \x01(\x0cR\x07\x63ommand\x12\x1d\n\npython_ver\x18\x02 \x01(\tR\tpythonVer"\x88\x01\n\x0e\x43ollectMetrics\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04name\x18\x02 \x01(\tR\x04name\x12\x33\n\x07metrics\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x07metrics"\x9a\x03\n\x05Parse\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x38\n\x06\x66ormat\x18\x02 \x01(\x0e\x32 .spark.connect.Parse.ParseFormatR\x06\x66ormat\x12\x34\n\x06schema\x18\x03 \x01(\x0b\x32\x17.spark.connect.DataTypeH\x00R\x06schema\x88\x01\x01\x12;\n\x07options\x18\x04 \x03(\x0b\x32!.spark.connect.Parse.OptionsEntryR\x07options\x1a:\n\x0cOptionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01"n\n\x0bParseFormat\x12\x1c\n\x18PARSE_FORMAT_UNSPECIFIED\x10\x00\x12\x14\n\x10PARSE_FORMAT_CSV\x10\x01\x12\x15\n\x11PARSE_FORMAT_JSON\x10\x02\x12\x14\n\x10PARSE_FORMAT_XML\x10\x03\x42\t\n\x07_schema"\xdb\x03\n\x08\x41sOfJoin\x12+\n\x04left\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04left\x12-\n\x05right\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\x05right\x12\x37\n\nleft_as_of\x18\x03 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x08leftAsOf\x12\x39\n\x0bright_as_of\x18\x04 \x01(\x0b\x32\x19.spark.connect.ExpressionR\trightAsOf\x12\x36\n\tjoin_expr\x18\x05 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x08joinExpr\x12#\n\rusing_columns\x18\x06 \x03(\tR\x0cusingColumns\x12\x1b\n\tjoin_type\x18\x07 \x01(\tR\x08joinType\x12\x37\n\ttolerance\x18\x08 \x01(\x0b\x32\x19.spark.connect.ExpressionR\ttolerance\x12.\n\x13\x61llow_exact_matches\x18\t \x01(\x08R\x11\x61llowExactMatches\x12\x1c\n\tdirection\x18\n \x01(\tR\tdirection"\xe6\x01\n\x0bLateralJoin\x12+\n\x04left\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04left\x12-\n\x05right\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\x05right\x12@\n\x0ejoin_condition\x18\x03 \x01(\x0b\x32\x19.spark.connect.ExpressionR\rjoinCondition\x12\x39\n\tjoin_type\x18\x04 \x01(\x0e\x32\x1c.spark.connect.Join.JoinTypeR\x08joinTypeB6\n\x1eorg.apache.spark.connect.protoP\x01Z\x12internal/generatedb\x06proto3' + b'\n\x1dspark/connect/relations.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1fspark/connect/expressions.proto\x1a\x19spark/connect/types.proto\x1a\x1bspark/connect/catalog.proto\x1a\x1aspark/connect/common.proto\x1a\x1dspark/connect/ml_common.proto"\xa1\x1f\n\x08Relation\x12\x35\n\x06\x63ommon\x18\x01 \x01(\x0b\x32\x1d.spark.connect.RelationCommonR\x06\x63ommon\x12)\n\x04read\x18\x02 \x01(\x0b\x32\x13.spark.connect.ReadH\x00R\x04read\x12\x32\n\x07project\x18\x03 \x01(\x0b\x32\x16.spark.connect.ProjectH\x00R\x07project\x12/\n\x06\x66ilter\x18\x04 \x01(\x0b\x32\x15.spark.connect.FilterH\x00R\x06\x66ilter\x12)\n\x04join\x18\x05 \x01(\x0b\x32\x13.spark.connect.JoinH\x00R\x04join\x12\x34\n\x06set_op\x18\x06 \x01(\x0b\x32\x1b.spark.connect.SetOperationH\x00R\x05setOp\x12)\n\x04sort\x18\x07 \x01(\x0b\x32\x13.spark.connect.SortH\x00R\x04sort\x12,\n\x05limit\x18\x08 \x01(\x0b\x32\x14.spark.connect.LimitH\x00R\x05limit\x12\x38\n\taggregate\x18\t \x01(\x0b\x32\x18.spark.connect.AggregateH\x00R\taggregate\x12&\n\x03sql\x18\n \x01(\x0b\x32\x12.spark.connect.SQLH\x00R\x03sql\x12\x45\n\x0elocal_relation\x18\x0b \x01(\x0b\x32\x1c.spark.connect.LocalRelationH\x00R\rlocalRelation\x12/\n\x06sample\x18\x0c \x01(\x0b\x32\x15.spark.connect.SampleH\x00R\x06sample\x12/\n\x06offset\x18\r \x01(\x0b\x32\x15.spark.connect.OffsetH\x00R\x06offset\x12>\n\x0b\x64\x65\x64uplicate\x18\x0e \x01(\x0b\x32\x1a.spark.connect.DeduplicateH\x00R\x0b\x64\x65\x64uplicate\x12,\n\x05range\x18\x0f \x01(\x0b\x32\x14.spark.connect.RangeH\x00R\x05range\x12\x45\n\x0esubquery_alias\x18\x10 \x01(\x0b\x32\x1c.spark.connect.SubqueryAliasH\x00R\rsubqueryAlias\x12>\n\x0brepartition\x18\x11 \x01(\x0b\x32\x1a.spark.connect.RepartitionH\x00R\x0brepartition\x12*\n\x05to_df\x18\x12 \x01(\x0b\x32\x13.spark.connect.ToDFH\x00R\x04toDf\x12U\n\x14with_columns_renamed\x18\x13 \x01(\x0b\x32!.spark.connect.WithColumnsRenamedH\x00R\x12withColumnsRenamed\x12<\n\x0bshow_string\x18\x14 \x01(\x0b\x32\x19.spark.connect.ShowStringH\x00R\nshowString\x12)\n\x04\x64rop\x18\x15 \x01(\x0b\x32\x13.spark.connect.DropH\x00R\x04\x64rop\x12)\n\x04tail\x18\x16 \x01(\x0b\x32\x13.spark.connect.TailH\x00R\x04tail\x12?\n\x0cwith_columns\x18\x17 \x01(\x0b\x32\x1a.spark.connect.WithColumnsH\x00R\x0bwithColumns\x12)\n\x04hint\x18\x18 \x01(\x0b\x32\x13.spark.connect.HintH\x00R\x04hint\x12\x32\n\x07unpivot\x18\x19 \x01(\x0b\x32\x16.spark.connect.UnpivotH\x00R\x07unpivot\x12\x36\n\tto_schema\x18\x1a \x01(\x0b\x32\x17.spark.connect.ToSchemaH\x00R\x08toSchema\x12\x64\n\x19repartition_by_expression\x18\x1b \x01(\x0b\x32&.spark.connect.RepartitionByExpressionH\x00R\x17repartitionByExpression\x12\x45\n\x0emap_partitions\x18\x1c \x01(\x0b\x32\x1c.spark.connect.MapPartitionsH\x00R\rmapPartitions\x12H\n\x0f\x63ollect_metrics\x18\x1d \x01(\x0b\x32\x1d.spark.connect.CollectMetricsH\x00R\x0e\x63ollectMetrics\x12,\n\x05parse\x18\x1e \x01(\x0b\x32\x14.spark.connect.ParseH\x00R\x05parse\x12\x36\n\tgroup_map\x18\x1f \x01(\x0b\x32\x17.spark.connect.GroupMapH\x00R\x08groupMap\x12=\n\x0c\x63o_group_map\x18 \x01(\x0b\x32\x19.spark.connect.CoGroupMapH\x00R\ncoGroupMap\x12\x45\n\x0ewith_watermark\x18! \x01(\x0b\x32\x1c.spark.connect.WithWatermarkH\x00R\rwithWatermark\x12\x63\n\x1a\x61pply_in_pandas_with_state\x18" \x01(\x0b\x32%.spark.connect.ApplyInPandasWithStateH\x00R\x16\x61pplyInPandasWithState\x12<\n\x0bhtml_string\x18# \x01(\x0b\x32\x19.spark.connect.HtmlStringH\x00R\nhtmlString\x12X\n\x15\x63\x61\x63hed_local_relation\x18$ \x01(\x0b\x32".spark.connect.CachedLocalRelationH\x00R\x13\x63\x61\x63hedLocalRelation\x12[\n\x16\x63\x61\x63hed_remote_relation\x18% \x01(\x0b\x32#.spark.connect.CachedRemoteRelationH\x00R\x14\x63\x61\x63hedRemoteRelation\x12\x8e\x01\n)common_inline_user_defined_table_function\x18& \x01(\x0b\x32\x33.spark.connect.CommonInlineUserDefinedTableFunctionH\x00R$commonInlineUserDefinedTableFunction\x12\x37\n\nas_of_join\x18\' \x01(\x0b\x32\x17.spark.connect.AsOfJoinH\x00R\x08\x61sOfJoin\x12\x85\x01\n&common_inline_user_defined_data_source\x18( \x01(\x0b\x32\x30.spark.connect.CommonInlineUserDefinedDataSourceH\x00R!commonInlineUserDefinedDataSource\x12\x45\n\x0ewith_relations\x18) \x01(\x0b\x32\x1c.spark.connect.WithRelationsH\x00R\rwithRelations\x12\x38\n\ttranspose\x18* \x01(\x0b\x32\x18.spark.connect.TransposeH\x00R\ttranspose\x12w\n unresolved_table_valued_function\x18+ \x01(\x0b\x32,.spark.connect.UnresolvedTableValuedFunctionH\x00R\x1dunresolvedTableValuedFunction\x12?\n\x0clateral_join\x18, \x01(\x0b\x32\x1a.spark.connect.LateralJoinH\x00R\x0blateralJoin\x12n\n\x1d\x63hunked_cached_local_relation\x18- \x01(\x0b\x32).spark.connect.ChunkedCachedLocalRelationH\x00R\x1a\x63hunkedCachedLocalRelation\x12K\n\x10relation_changes\x18. \x01(\x0b\x32\x1e.spark.connect.RelationChangesH\x00R\x0frelationChanges\x12\x46\n\x0fnearest_by_join\x18/ \x01(\x0b\x32\x1c.spark.connect.NearestByJoinH\x00R\rnearestByJoin\x12\x30\n\x07\x66ill_na\x18Z \x01(\x0b\x32\x15.spark.connect.NAFillH\x00R\x06\x66illNa\x12\x30\n\x07\x64rop_na\x18[ \x01(\x0b\x32\x15.spark.connect.NADropH\x00R\x06\x64ropNa\x12\x34\n\x07replace\x18\\ \x01(\x0b\x32\x18.spark.connect.NAReplaceH\x00R\x07replace\x12\x36\n\x07summary\x18\x64 \x01(\x0b\x32\x1a.spark.connect.StatSummaryH\x00R\x07summary\x12\x39\n\x08\x63rosstab\x18\x65 \x01(\x0b\x32\x1b.spark.connect.StatCrosstabH\x00R\x08\x63rosstab\x12\x39\n\x08\x64\x65scribe\x18\x66 \x01(\x0b\x32\x1b.spark.connect.StatDescribeH\x00R\x08\x64\x65scribe\x12*\n\x03\x63ov\x18g \x01(\x0b\x32\x16.spark.connect.StatCovH\x00R\x03\x63ov\x12-\n\x04\x63orr\x18h \x01(\x0b\x32\x17.spark.connect.StatCorrH\x00R\x04\x63orr\x12L\n\x0f\x61pprox_quantile\x18i \x01(\x0b\x32!.spark.connect.StatApproxQuantileH\x00R\x0e\x61pproxQuantile\x12=\n\nfreq_items\x18j \x01(\x0b\x32\x1c.spark.connect.StatFreqItemsH\x00R\tfreqItems\x12:\n\tsample_by\x18k \x01(\x0b\x32\x1b.spark.connect.StatSampleByH\x00R\x08sampleBy\x12\x33\n\x07\x63\x61talog\x18\xc8\x01 \x01(\x0b\x32\x16.spark.connect.CatalogH\x00R\x07\x63\x61talog\x12=\n\x0bml_relation\x18\xac\x02 \x01(\x0b\x32\x19.spark.connect.MlRelationH\x00R\nmlRelation\x12\x35\n\textension\x18\xe6\x07 \x01(\x0b\x32\x14.google.protobuf.AnyH\x00R\textension\x12\x33\n\x07unknown\x18\xe7\x07 \x01(\x0b\x32\x16.spark.connect.UnknownH\x00R\x07unknownB\n\n\x08rel_type"\xe4\x03\n\nMlRelation\x12\x43\n\ttransform\x18\x01 \x01(\x0b\x32#.spark.connect.MlRelation.TransformH\x00R\ttransform\x12,\n\x05\x66\x65tch\x18\x02 \x01(\x0b\x32\x14.spark.connect.FetchH\x00R\x05\x66\x65tch\x12P\n\x15model_summary_dataset\x18\x03 \x01(\x0b\x32\x17.spark.connect.RelationH\x01R\x13modelSummaryDataset\x88\x01\x01\x1a\xeb\x01\n\tTransform\x12\x33\n\x07obj_ref\x18\x01 \x01(\x0b\x32\x18.spark.connect.ObjectRefH\x00R\x06objRef\x12=\n\x0btransformer\x18\x02 \x01(\x0b\x32\x19.spark.connect.MlOperatorH\x00R\x0btransformer\x12-\n\x05input\x18\x03 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12/\n\x06params\x18\x04 \x01(\x0b\x32\x17.spark.connect.MlParamsR\x06paramsB\n\n\x08operatorB\t\n\x07ml_typeB\x18\n\x16_model_summary_dataset"\xcb\x02\n\x05\x46\x65tch\x12\x31\n\x07obj_ref\x18\x01 \x01(\x0b\x32\x18.spark.connect.ObjectRefR\x06objRef\x12\x35\n\x07methods\x18\x02 \x03(\x0b\x32\x1b.spark.connect.Fetch.MethodR\x07methods\x1a\xd7\x01\n\x06Method\x12\x16\n\x06method\x18\x01 \x01(\tR\x06method\x12\x34\n\x04\x61rgs\x18\x02 \x03(\x0b\x32 .spark.connect.Fetch.Method.ArgsR\x04\x61rgs\x1a\x7f\n\x04\x41rgs\x12\x39\n\x05param\x18\x01 \x01(\x0b\x32!.spark.connect.Expression.LiteralH\x00R\x05param\x12/\n\x05input\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationH\x00R\x05inputB\x0b\n\targs_type"\t\n\x07Unknown"\x8e\x01\n\x0eRelationCommon\x12#\n\x0bsource_info\x18\x01 \x01(\tB\x02\x18\x01R\nsourceInfo\x12\x1c\n\x07plan_id\x18\x02 \x01(\x03H\x00R\x06planId\x88\x01\x01\x12-\n\x06origin\x18\x03 \x01(\x0b\x32\x15.spark.connect.OriginR\x06originB\n\n\x08_plan_id"\xde\x03\n\x03SQL\x12\x14\n\x05query\x18\x01 \x01(\tR\x05query\x12\x34\n\x04\x61rgs\x18\x02 \x03(\x0b\x32\x1c.spark.connect.SQL.ArgsEntryB\x02\x18\x01R\x04\x61rgs\x12@\n\x08pos_args\x18\x03 \x03(\x0b\x32!.spark.connect.Expression.LiteralB\x02\x18\x01R\x07posArgs\x12O\n\x0fnamed_arguments\x18\x04 \x03(\x0b\x32&.spark.connect.SQL.NamedArgumentsEntryR\x0enamedArguments\x12>\n\rpos_arguments\x18\x05 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0cposArguments\x1aZ\n\tArgsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x37\n\x05value\x18\x02 \x01(\x0b\x32!.spark.connect.Expression.LiteralR\x05value:\x02\x38\x01\x1a\\\n\x13NamedArgumentsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12/\n\x05value\x18\x02 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x05value:\x02\x38\x01"u\n\rWithRelations\x12+\n\x04root\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04root\x12\x37\n\nreferences\x18\x02 \x03(\x0b\x32\x17.spark.connect.RelationR\nreferences"\xcd\x05\n\x04Read\x12\x41\n\x0bnamed_table\x18\x01 \x01(\x0b\x32\x1e.spark.connect.Read.NamedTableH\x00R\nnamedTable\x12\x41\n\x0b\x64\x61ta_source\x18\x02 \x01(\x0b\x32\x1e.spark.connect.Read.DataSourceH\x00R\ndataSource\x12!\n\x0cis_streaming\x18\x03 \x01(\x08R\x0bisStreaming\x1a\xc0\x01\n\nNamedTable\x12/\n\x13unparsed_identifier\x18\x01 \x01(\tR\x12unparsedIdentifier\x12\x45\n\x07options\x18\x02 \x03(\x0b\x32+.spark.connect.Read.NamedTable.OptionsEntryR\x07options\x1a:\n\x0cOptionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1a\xcb\x02\n\nDataSource\x12\x1b\n\x06\x66ormat\x18\x01 \x01(\tH\x00R\x06\x66ormat\x88\x01\x01\x12\x1b\n\x06schema\x18\x02 \x01(\tH\x01R\x06schema\x88\x01\x01\x12\x45\n\x07options\x18\x03 \x03(\x0b\x32+.spark.connect.Read.DataSource.OptionsEntryR\x07options\x12\x14\n\x05paths\x18\x04 \x03(\tR\x05paths\x12\x1e\n\npredicates\x18\x05 \x03(\tR\npredicates\x12$\n\x0bsource_name\x18\x06 \x01(\tH\x02R\nsourceName\x88\x01\x01\x1a:\n\x0cOptionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x42\t\n\x07_formatB\t\n\x07_schemaB\x0e\n\x0c_source_nameB\x0b\n\tread_type"\xe8\x01\n\x0fRelationChanges\x12/\n\x13unparsed_identifier\x18\x01 \x01(\tR\x12unparsedIdentifier\x12\x45\n\x07options\x18\x02 \x03(\x0b\x32+.spark.connect.RelationChanges.OptionsEntryR\x07options\x12!\n\x0cis_streaming\x18\x03 \x01(\x08R\x0bisStreaming\x1a:\n\x0cOptionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01"u\n\x07Project\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12;\n\x0b\x65xpressions\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0b\x65xpressions"p\n\x06\x46ilter\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x37\n\tcondition\x18\x02 \x01(\x0b\x32\x19.spark.connect.ExpressionR\tcondition"\x95\x05\n\x04Join\x12+\n\x04left\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04left\x12-\n\x05right\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\x05right\x12@\n\x0ejoin_condition\x18\x03 \x01(\x0b\x32\x19.spark.connect.ExpressionR\rjoinCondition\x12\x39\n\tjoin_type\x18\x04 \x01(\x0e\x32\x1c.spark.connect.Join.JoinTypeR\x08joinType\x12#\n\rusing_columns\x18\x05 \x03(\tR\x0cusingColumns\x12K\n\x0ejoin_data_type\x18\x06 \x01(\x0b\x32 .spark.connect.Join.JoinDataTypeH\x00R\x0cjoinDataType\x88\x01\x01\x1a\\\n\x0cJoinDataType\x12$\n\x0eis_left_struct\x18\x01 \x01(\x08R\x0cisLeftStruct\x12&\n\x0fis_right_struct\x18\x02 \x01(\x08R\risRightStruct"\xd0\x01\n\x08JoinType\x12\x19\n\x15JOIN_TYPE_UNSPECIFIED\x10\x00\x12\x13\n\x0fJOIN_TYPE_INNER\x10\x01\x12\x18\n\x14JOIN_TYPE_FULL_OUTER\x10\x02\x12\x18\n\x14JOIN_TYPE_LEFT_OUTER\x10\x03\x12\x19\n\x15JOIN_TYPE_RIGHT_OUTER\x10\x04\x12\x17\n\x13JOIN_TYPE_LEFT_ANTI\x10\x05\x12\x17\n\x13JOIN_TYPE_LEFT_SEMI\x10\x06\x12\x13\n\x0fJOIN_TYPE_CROSS\x10\x07\x42\x11\n\x0f_join_data_type"\xdf\x03\n\x0cSetOperation\x12\x36\n\nleft_input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\tleftInput\x12\x38\n\x0bright_input\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\nrightInput\x12\x45\n\x0bset_op_type\x18\x03 \x01(\x0e\x32%.spark.connect.SetOperation.SetOpTypeR\tsetOpType\x12\x1a\n\x06is_all\x18\x04 \x01(\x08H\x00R\x05isAll\x88\x01\x01\x12\x1c\n\x07\x62y_name\x18\x05 \x01(\x08H\x01R\x06\x62yName\x88\x01\x01\x12\x37\n\x15\x61llow_missing_columns\x18\x06 \x01(\x08H\x02R\x13\x61llowMissingColumns\x88\x01\x01"r\n\tSetOpType\x12\x1b\n\x17SET_OP_TYPE_UNSPECIFIED\x10\x00\x12\x19\n\x15SET_OP_TYPE_INTERSECT\x10\x01\x12\x15\n\x11SET_OP_TYPE_UNION\x10\x02\x12\x16\n\x12SET_OP_TYPE_EXCEPT\x10\x03\x42\t\n\x07_is_allB\n\n\x08_by_nameB\x18\n\x16_allow_missing_columns"L\n\x05Limit\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x14\n\x05limit\x18\x02 \x01(\x05R\x05limit"O\n\x06Offset\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x16\n\x06offset\x18\x02 \x01(\x05R\x06offset"K\n\x04Tail\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x14\n\x05limit\x18\x02 \x01(\x05R\x05limit"\xfe\x05\n\tAggregate\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x41\n\ngroup_type\x18\x02 \x01(\x0e\x32".spark.connect.Aggregate.GroupTypeR\tgroupType\x12L\n\x14grouping_expressions\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x13groupingExpressions\x12N\n\x15\x61ggregate_expressions\x18\x04 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x14\x61ggregateExpressions\x12\x34\n\x05pivot\x18\x05 \x01(\x0b\x32\x1e.spark.connect.Aggregate.PivotR\x05pivot\x12J\n\rgrouping_sets\x18\x06 \x03(\x0b\x32%.spark.connect.Aggregate.GroupingSetsR\x0cgroupingSets\x1ao\n\x05Pivot\x12+\n\x03\x63ol\x18\x01 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x03\x63ol\x12\x39\n\x06values\x18\x02 \x03(\x0b\x32!.spark.connect.Expression.LiteralR\x06values\x1aL\n\x0cGroupingSets\x12<\n\x0cgrouping_set\x18\x01 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0bgroupingSet"\x9f\x01\n\tGroupType\x12\x1a\n\x16GROUP_TYPE_UNSPECIFIED\x10\x00\x12\x16\n\x12GROUP_TYPE_GROUPBY\x10\x01\x12\x15\n\x11GROUP_TYPE_ROLLUP\x10\x02\x12\x13\n\x0fGROUP_TYPE_CUBE\x10\x03\x12\x14\n\x10GROUP_TYPE_PIVOT\x10\x04\x12\x1c\n\x18GROUP_TYPE_GROUPING_SETS\x10\x05"\xa0\x01\n\x04Sort\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x39\n\x05order\x18\x02 \x03(\x0b\x32#.spark.connect.Expression.SortOrderR\x05order\x12 \n\tis_global\x18\x03 \x01(\x08H\x00R\x08isGlobal\x88\x01\x01\x42\x0c\n\n_is_global"\x8d\x01\n\x04\x44rop\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x33\n\x07\x63olumns\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x07\x63olumns\x12!\n\x0c\x63olumn_names\x18\x03 \x03(\tR\x0b\x63olumnNames"\xf0\x01\n\x0b\x44\x65\x64uplicate\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12!\n\x0c\x63olumn_names\x18\x02 \x03(\tR\x0b\x63olumnNames\x12\x32\n\x13\x61ll_columns_as_keys\x18\x03 \x01(\x08H\x00R\x10\x61llColumnsAsKeys\x88\x01\x01\x12.\n\x10within_watermark\x18\x04 \x01(\x08H\x01R\x0fwithinWatermark\x88\x01\x01\x42\x16\n\x14_all_columns_as_keysB\x13\n\x11_within_watermark"Y\n\rLocalRelation\x12\x17\n\x04\x64\x61ta\x18\x01 \x01(\x0cH\x00R\x04\x64\x61ta\x88\x01\x01\x12\x1b\n\x06schema\x18\x02 \x01(\tH\x01R\x06schema\x88\x01\x01\x42\x07\n\x05_dataB\t\n\x07_schema"H\n\x13\x43\x61\x63hedLocalRelation\x12\x12\n\x04hash\x18\x03 \x01(\tR\x04hashJ\x04\x08\x01\x10\x02J\x04\x08\x02\x10\x03R\x06userIdR\tsessionId"p\n\x1a\x43hunkedCachedLocalRelation\x12\x1e\n\ndataHashes\x18\x01 \x03(\tR\ndataHashes\x12#\n\nschemaHash\x18\x02 \x01(\tH\x00R\nschemaHash\x88\x01\x01\x42\r\n\x0b_schemaHash"7\n\x14\x43\x61\x63hedRemoteRelation\x12\x1f\n\x0brelation_id\x18\x01 \x01(\tR\nrelationId"\x91\x02\n\x06Sample\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x1f\n\x0blower_bound\x18\x02 \x01(\x01R\nlowerBound\x12\x1f\n\x0bupper_bound\x18\x03 \x01(\x01R\nupperBound\x12.\n\x10with_replacement\x18\x04 \x01(\x08H\x00R\x0fwithReplacement\x88\x01\x01\x12\x17\n\x04seed\x18\x05 \x01(\x03H\x01R\x04seed\x88\x01\x01\x12/\n\x13\x64\x65terministic_order\x18\x06 \x01(\x08R\x12\x64\x65terministicOrderB\x13\n\x11_with_replacementB\x07\n\x05_seed"\x91\x01\n\x05Range\x12\x19\n\x05start\x18\x01 \x01(\x03H\x00R\x05start\x88\x01\x01\x12\x10\n\x03\x65nd\x18\x02 \x01(\x03R\x03\x65nd\x12\x12\n\x04step\x18\x03 \x01(\x03R\x04step\x12*\n\x0enum_partitions\x18\x04 \x01(\x05H\x01R\rnumPartitions\x88\x01\x01\x42\x08\n\x06_startB\x11\n\x0f_num_partitions"r\n\rSubqueryAlias\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x14\n\x05\x61lias\x18\x02 \x01(\tR\x05\x61lias\x12\x1c\n\tqualifier\x18\x03 \x03(\tR\tqualifier"\x8e\x01\n\x0bRepartition\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12%\n\x0enum_partitions\x18\x02 \x01(\x05R\rnumPartitions\x12\x1d\n\x07shuffle\x18\x03 \x01(\x08H\x00R\x07shuffle\x88\x01\x01\x42\n\n\x08_shuffle"\x8e\x01\n\nShowString\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x19\n\x08num_rows\x18\x02 \x01(\x05R\x07numRows\x12\x1a\n\x08truncate\x18\x03 \x01(\x05R\x08truncate\x12\x1a\n\x08vertical\x18\x04 \x01(\x08R\x08vertical"r\n\nHtmlString\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x19\n\x08num_rows\x18\x02 \x01(\x05R\x07numRows\x12\x1a\n\x08truncate\x18\x03 \x01(\x05R\x08truncate"\\\n\x0bStatSummary\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x1e\n\nstatistics\x18\x02 \x03(\tR\nstatistics"Q\n\x0cStatDescribe\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols"e\n\x0cStatCrosstab\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ol1\x18\x02 \x01(\tR\x04\x63ol1\x12\x12\n\x04\x63ol2\x18\x03 \x01(\tR\x04\x63ol2"`\n\x07StatCov\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ol1\x18\x02 \x01(\tR\x04\x63ol1\x12\x12\n\x04\x63ol2\x18\x03 \x01(\tR\x04\x63ol2"\x89\x01\n\x08StatCorr\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ol1\x18\x02 \x01(\tR\x04\x63ol1\x12\x12\n\x04\x63ol2\x18\x03 \x01(\tR\x04\x63ol2\x12\x1b\n\x06method\x18\x04 \x01(\tH\x00R\x06method\x88\x01\x01\x42\t\n\x07_method"\xa4\x01\n\x12StatApproxQuantile\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12$\n\rprobabilities\x18\x03 \x03(\x01R\rprobabilities\x12%\n\x0erelative_error\x18\x04 \x01(\x01R\rrelativeError"}\n\rStatFreqItems\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12\x1d\n\x07support\x18\x03 \x01(\x01H\x00R\x07support\x88\x01\x01\x42\n\n\x08_support"\xb5\x02\n\x0cStatSampleBy\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12+\n\x03\x63ol\x18\x02 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x03\x63ol\x12\x42\n\tfractions\x18\x03 \x03(\x0b\x32$.spark.connect.StatSampleBy.FractionR\tfractions\x12\x17\n\x04seed\x18\x05 \x01(\x03H\x00R\x04seed\x88\x01\x01\x1a\x63\n\x08\x46raction\x12;\n\x07stratum\x18\x01 \x01(\x0b\x32!.spark.connect.Expression.LiteralR\x07stratum\x12\x1a\n\x08\x66raction\x18\x02 \x01(\x01R\x08\x66ractionB\x07\n\x05_seed"\x86\x01\n\x06NAFill\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12\x39\n\x06values\x18\x03 \x03(\x0b\x32!.spark.connect.Expression.LiteralR\x06values"\x86\x01\n\x06NADrop\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12\'\n\rmin_non_nulls\x18\x03 \x01(\x05H\x00R\x0bminNonNulls\x88\x01\x01\x42\x10\n\x0e_min_non_nulls"\xa8\x02\n\tNAReplace\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04\x63ols\x18\x02 \x03(\tR\x04\x63ols\x12H\n\x0creplacements\x18\x03 \x03(\x0b\x32$.spark.connect.NAReplace.ReplacementR\x0creplacements\x1a\x8d\x01\n\x0bReplacement\x12>\n\told_value\x18\x01 \x01(\x0b\x32!.spark.connect.Expression.LiteralR\x08oldValue\x12>\n\tnew_value\x18\x02 \x01(\x0b\x32!.spark.connect.Expression.LiteralR\x08newValue"X\n\x04ToDF\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12!\n\x0c\x63olumn_names\x18\x02 \x03(\tR\x0b\x63olumnNames"\xfe\x02\n\x12WithColumnsRenamed\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12i\n\x12rename_columns_map\x18\x02 \x03(\x0b\x32\x37.spark.connect.WithColumnsRenamed.RenameColumnsMapEntryB\x02\x18\x01R\x10renameColumnsMap\x12\x42\n\x07renames\x18\x03 \x03(\x0b\x32(.spark.connect.WithColumnsRenamed.RenameR\x07renames\x1a\x43\n\x15RenameColumnsMapEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1a\x45\n\x06Rename\x12\x19\n\x08\x63ol_name\x18\x01 \x01(\tR\x07\x63olName\x12 \n\x0cnew_col_name\x18\x02 \x01(\tR\nnewColName"w\n\x0bWithColumns\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x39\n\x07\x61liases\x18\x02 \x03(\x0b\x32\x1f.spark.connect.Expression.AliasR\x07\x61liases"\x86\x01\n\rWithWatermark\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x1d\n\nevent_time\x18\x02 \x01(\tR\teventTime\x12\'\n\x0f\x64\x65lay_threshold\x18\x03 \x01(\tR\x0e\x64\x65layThreshold"\x84\x01\n\x04Hint\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04name\x18\x02 \x01(\tR\x04name\x12\x39\n\nparameters\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\nparameters"\xc7\x02\n\x07Unpivot\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12+\n\x03ids\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x03ids\x12:\n\x06values\x18\x03 \x01(\x0b\x32\x1d.spark.connect.Unpivot.ValuesH\x00R\x06values\x88\x01\x01\x12\x30\n\x14variable_column_name\x18\x04 \x01(\tR\x12variableColumnName\x12*\n\x11value_column_name\x18\x05 \x01(\tR\x0fvalueColumnName\x1a;\n\x06Values\x12\x31\n\x06values\x18\x01 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x06valuesB\t\n\x07_values"z\n\tTranspose\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12>\n\rindex_columns\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0cindexColumns"}\n\x1dUnresolvedTableValuedFunction\x12#\n\rfunction_name\x18\x01 \x01(\tR\x0c\x66unctionName\x12\x37\n\targuments\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\targuments"j\n\x08ToSchema\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12/\n\x06schema\x18\x02 \x01(\x0b\x32\x17.spark.connect.DataTypeR\x06schema"\xcb\x01\n\x17RepartitionByExpression\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x42\n\x0fpartition_exprs\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x0epartitionExprs\x12*\n\x0enum_partitions\x18\x03 \x01(\x05H\x00R\rnumPartitions\x88\x01\x01\x42\x11\n\x0f_num_partitions"\xe8\x01\n\rMapPartitions\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x42\n\x04\x66unc\x18\x02 \x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionR\x04\x66unc\x12"\n\nis_barrier\x18\x03 \x01(\x08H\x00R\tisBarrier\x88\x01\x01\x12"\n\nprofile_id\x18\x04 \x01(\x05H\x01R\tprofileId\x88\x01\x01\x42\r\n\x0b_is_barrierB\r\n\x0b_profile_id"\xd2\x06\n\x08GroupMap\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12L\n\x14grouping_expressions\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x13groupingExpressions\x12\x42\n\x04\x66unc\x18\x03 \x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionR\x04\x66unc\x12J\n\x13sorting_expressions\x18\x04 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x12sortingExpressions\x12<\n\rinitial_input\x18\x05 \x01(\x0b\x32\x17.spark.connect.RelationR\x0cinitialInput\x12[\n\x1cinitial_grouping_expressions\x18\x06 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x1ainitialGroupingExpressions\x12;\n\x18is_map_groups_with_state\x18\x07 \x01(\x08H\x00R\x14isMapGroupsWithState\x88\x01\x01\x12$\n\x0boutput_mode\x18\x08 \x01(\tH\x01R\noutputMode\x88\x01\x01\x12&\n\x0ctimeout_conf\x18\t \x01(\tH\x02R\x0btimeoutConf\x88\x01\x01\x12?\n\x0cstate_schema\x18\n \x01(\x0b\x32\x17.spark.connect.DataTypeH\x03R\x0bstateSchema\x88\x01\x01\x12\x65\n\x19transform_with_state_info\x18\x0b \x01(\x0b\x32%.spark.connect.TransformWithStateInfoH\x04R\x16transformWithStateInfo\x88\x01\x01\x42\x1b\n\x19_is_map_groups_with_stateB\x0e\n\x0c_output_modeB\x0f\n\r_timeout_confB\x0f\n\r_state_schemaB\x1c\n\x1a_transform_with_state_info"\xdf\x01\n\x16TransformWithStateInfo\x12\x1b\n\ttime_mode\x18\x01 \x01(\tR\x08timeMode\x12\x38\n\x16\x65vent_time_column_name\x18\x02 \x01(\tH\x00R\x13\x65ventTimeColumnName\x88\x01\x01\x12\x41\n\routput_schema\x18\x03 \x01(\x0b\x32\x17.spark.connect.DataTypeH\x01R\x0coutputSchema\x88\x01\x01\x42\x19\n\x17_event_time_column_nameB\x10\n\x0e_output_schema"\x8e\x04\n\nCoGroupMap\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12W\n\x1ainput_grouping_expressions\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x18inputGroupingExpressions\x12-\n\x05other\x18\x03 \x01(\x0b\x32\x17.spark.connect.RelationR\x05other\x12W\n\x1aother_grouping_expressions\x18\x04 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x18otherGroupingExpressions\x12\x42\n\x04\x66unc\x18\x05 \x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionR\x04\x66unc\x12U\n\x19input_sorting_expressions\x18\x06 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x17inputSortingExpressions\x12U\n\x19other_sorting_expressions\x18\x07 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x17otherSortingExpressions"\xe5\x02\n\x16\x41pplyInPandasWithState\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12L\n\x14grouping_expressions\x18\x02 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x13groupingExpressions\x12\x42\n\x04\x66unc\x18\x03 \x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionR\x04\x66unc\x12#\n\routput_schema\x18\x04 \x01(\tR\x0coutputSchema\x12!\n\x0cstate_schema\x18\x05 \x01(\tR\x0bstateSchema\x12\x1f\n\x0boutput_mode\x18\x06 \x01(\tR\noutputMode\x12!\n\x0ctimeout_conf\x18\x07 \x01(\tR\x0btimeoutConf"\xf4\x01\n$CommonInlineUserDefinedTableFunction\x12#\n\rfunction_name\x18\x01 \x01(\tR\x0c\x66unctionName\x12$\n\rdeterministic\x18\x02 \x01(\x08R\rdeterministic\x12\x37\n\targuments\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\targuments\x12<\n\x0bpython_udtf\x18\x04 \x01(\x0b\x32\x19.spark.connect.PythonUDTFH\x00R\npythonUdtfB\n\n\x08\x66unction"\xb1\x01\n\nPythonUDTF\x12=\n\x0breturn_type\x18\x01 \x01(\x0b\x32\x17.spark.connect.DataTypeH\x00R\nreturnType\x88\x01\x01\x12\x1b\n\teval_type\x18\x02 \x01(\x05R\x08\x65valType\x12\x18\n\x07\x63ommand\x18\x03 \x01(\x0cR\x07\x63ommand\x12\x1d\n\npython_ver\x18\x04 \x01(\tR\tpythonVerB\x0e\n\x0c_return_type"\x97\x01\n!CommonInlineUserDefinedDataSource\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12O\n\x12python_data_source\x18\x02 \x01(\x0b\x32\x1f.spark.connect.PythonDataSourceH\x00R\x10pythonDataSourceB\r\n\x0b\x64\x61ta_source"K\n\x10PythonDataSource\x12\x18\n\x07\x63ommand\x18\x01 \x01(\x0cR\x07\x63ommand\x12\x1d\n\npython_ver\x18\x02 \x01(\tR\tpythonVer"\x88\x01\n\x0e\x43ollectMetrics\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x12\n\x04name\x18\x02 \x01(\tR\x04name\x12\x33\n\x07metrics\x18\x03 \x03(\x0b\x32\x19.spark.connect.ExpressionR\x07metrics"\x9a\x03\n\x05Parse\x12-\n\x05input\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x05input\x12\x38\n\x06\x66ormat\x18\x02 \x01(\x0e\x32 .spark.connect.Parse.ParseFormatR\x06\x66ormat\x12\x34\n\x06schema\x18\x03 \x01(\x0b\x32\x17.spark.connect.DataTypeH\x00R\x06schema\x88\x01\x01\x12;\n\x07options\x18\x04 \x03(\x0b\x32!.spark.connect.Parse.OptionsEntryR\x07options\x1a:\n\x0cOptionsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01"n\n\x0bParseFormat\x12\x1c\n\x18PARSE_FORMAT_UNSPECIFIED\x10\x00\x12\x14\n\x10PARSE_FORMAT_CSV\x10\x01\x12\x15\n\x11PARSE_FORMAT_JSON\x10\x02\x12\x14\n\x10PARSE_FORMAT_XML\x10\x03\x42\t\n\x07_schema"\xdb\x03\n\x08\x41sOfJoin\x12+\n\x04left\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04left\x12-\n\x05right\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\x05right\x12\x37\n\nleft_as_of\x18\x03 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x08leftAsOf\x12\x39\n\x0bright_as_of\x18\x04 \x01(\x0b\x32\x19.spark.connect.ExpressionR\trightAsOf\x12\x36\n\tjoin_expr\x18\x05 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x08joinExpr\x12#\n\rusing_columns\x18\x06 \x03(\tR\x0cusingColumns\x12\x1b\n\tjoin_type\x18\x07 \x01(\tR\x08joinType\x12\x37\n\ttolerance\x18\x08 \x01(\x0b\x32\x19.spark.connect.ExpressionR\ttolerance\x12.\n\x13\x61llow_exact_matches\x18\t \x01(\x08R\x11\x61llowExactMatches\x12\x1c\n\tdirection\x18\n \x01(\tR\tdirection"\xe6\x01\n\x0bLateralJoin\x12+\n\x04left\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04left\x12-\n\x05right\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\x05right\x12@\n\x0ejoin_condition\x18\x03 \x01(\x0b\x32\x19.spark.connect.ExpressionR\rjoinCondition\x12\x39\n\tjoin_type\x18\x04 \x01(\x0e\x32\x1c.spark.connect.Join.JoinTypeR\x08joinType"\xa5\x02\n\rNearestByJoin\x12+\n\x04left\x18\x01 \x01(\x0b\x32\x17.spark.connect.RelationR\x04left\x12-\n\x05right\x18\x02 \x01(\x0b\x32\x17.spark.connect.RelationR\x05right\x12H\n\x12ranking_expression\x18\x03 \x01(\x0b\x32\x19.spark.connect.ExpressionR\x11rankingExpression\x12\x1f\n\x0bnum_results\x18\x04 \x01(\x05R\nnumResults\x12\x1b\n\tjoin_type\x18\x05 \x01(\tR\x08joinType\x12\x12\n\x04mode\x18\x06 \x01(\tR\x04mode\x12\x1c\n\tdirection\x18\x07 \x01(\tR\tdirectionB6\n\x1eorg.apache.spark.connect.protoP\x01Z\x12internal/generatedb\x06proto3' ) _globals = globals() @@ -82,177 +82,179 @@ _globals["_PARSE_OPTIONSENTRY"]._loaded_options = None _globals["_PARSE_OPTIONSENTRY"]._serialized_options = b"8\001" _globals["_RELATION"]._serialized_start = 224 - _globals["_RELATION"]._serialized_end = 4153 - _globals["_MLRELATION"]._serialized_start = 4156 - _globals["_MLRELATION"]._serialized_end = 4640 - _globals["_MLRELATION_TRANSFORM"]._serialized_start = 4368 - _globals["_MLRELATION_TRANSFORM"]._serialized_end = 4603 - _globals["_FETCH"]._serialized_start = 4643 - _globals["_FETCH"]._serialized_end = 4974 - _globals["_FETCH_METHOD"]._serialized_start = 4759 - _globals["_FETCH_METHOD"]._serialized_end = 4974 - _globals["_FETCH_METHOD_ARGS"]._serialized_start = 4847 - _globals["_FETCH_METHOD_ARGS"]._serialized_end = 4974 - _globals["_UNKNOWN"]._serialized_start = 4976 - _globals["_UNKNOWN"]._serialized_end = 4985 - _globals["_RELATIONCOMMON"]._serialized_start = 4988 - _globals["_RELATIONCOMMON"]._serialized_end = 5130 - _globals["_SQL"]._serialized_start = 5133 - _globals["_SQL"]._serialized_end = 5611 - _globals["_SQL_ARGSENTRY"]._serialized_start = 5427 - _globals["_SQL_ARGSENTRY"]._serialized_end = 5517 - _globals["_SQL_NAMEDARGUMENTSENTRY"]._serialized_start = 5519 - _globals["_SQL_NAMEDARGUMENTSENTRY"]._serialized_end = 5611 - _globals["_WITHRELATIONS"]._serialized_start = 5613 - _globals["_WITHRELATIONS"]._serialized_end = 5730 - _globals["_READ"]._serialized_start = 5733 - _globals["_READ"]._serialized_end = 6450 - _globals["_READ_NAMEDTABLE"]._serialized_start = 5911 - _globals["_READ_NAMEDTABLE"]._serialized_end = 6103 - _globals["_READ_NAMEDTABLE_OPTIONSENTRY"]._serialized_start = 6045 - _globals["_READ_NAMEDTABLE_OPTIONSENTRY"]._serialized_end = 6103 - _globals["_READ_DATASOURCE"]._serialized_start = 6106 - _globals["_READ_DATASOURCE"]._serialized_end = 6437 - _globals["_READ_DATASOURCE_OPTIONSENTRY"]._serialized_start = 6045 - _globals["_READ_DATASOURCE_OPTIONSENTRY"]._serialized_end = 6103 - _globals["_RELATIONCHANGES"]._serialized_start = 6453 - _globals["_RELATIONCHANGES"]._serialized_end = 6685 - _globals["_RELATIONCHANGES_OPTIONSENTRY"]._serialized_start = 6045 - _globals["_RELATIONCHANGES_OPTIONSENTRY"]._serialized_end = 6103 - _globals["_PROJECT"]._serialized_start = 6687 - _globals["_PROJECT"]._serialized_end = 6804 - _globals["_FILTER"]._serialized_start = 6806 - _globals["_FILTER"]._serialized_end = 6918 - _globals["_JOIN"]._serialized_start = 6921 - _globals["_JOIN"]._serialized_end = 7582 - _globals["_JOIN_JOINDATATYPE"]._serialized_start = 7260 - _globals["_JOIN_JOINDATATYPE"]._serialized_end = 7352 - _globals["_JOIN_JOINTYPE"]._serialized_start = 7355 - _globals["_JOIN_JOINTYPE"]._serialized_end = 7563 - _globals["_SETOPERATION"]._serialized_start = 7585 - _globals["_SETOPERATION"]._serialized_end = 8064 - _globals["_SETOPERATION_SETOPTYPE"]._serialized_start = 7901 - _globals["_SETOPERATION_SETOPTYPE"]._serialized_end = 8015 - _globals["_LIMIT"]._serialized_start = 8066 - _globals["_LIMIT"]._serialized_end = 8142 - _globals["_OFFSET"]._serialized_start = 8144 - _globals["_OFFSET"]._serialized_end = 8223 - _globals["_TAIL"]._serialized_start = 8225 - _globals["_TAIL"]._serialized_end = 8300 - _globals["_AGGREGATE"]._serialized_start = 8303 - _globals["_AGGREGATE"]._serialized_end = 9069 - _globals["_AGGREGATE_PIVOT"]._serialized_start = 8718 - _globals["_AGGREGATE_PIVOT"]._serialized_end = 8829 - _globals["_AGGREGATE_GROUPINGSETS"]._serialized_start = 8831 - _globals["_AGGREGATE_GROUPINGSETS"]._serialized_end = 8907 - _globals["_AGGREGATE_GROUPTYPE"]._serialized_start = 8910 - _globals["_AGGREGATE_GROUPTYPE"]._serialized_end = 9069 - _globals["_SORT"]._serialized_start = 9072 - _globals["_SORT"]._serialized_end = 9232 - _globals["_DROP"]._serialized_start = 9235 - _globals["_DROP"]._serialized_end = 9376 - _globals["_DEDUPLICATE"]._serialized_start = 9379 - _globals["_DEDUPLICATE"]._serialized_end = 9619 - _globals["_LOCALRELATION"]._serialized_start = 9621 - _globals["_LOCALRELATION"]._serialized_end = 9710 - _globals["_CACHEDLOCALRELATION"]._serialized_start = 9712 - _globals["_CACHEDLOCALRELATION"]._serialized_end = 9784 - _globals["_CHUNKEDCACHEDLOCALRELATION"]._serialized_start = 9786 - _globals["_CHUNKEDCACHEDLOCALRELATION"]._serialized_end = 9898 - _globals["_CACHEDREMOTERELATION"]._serialized_start = 9900 - _globals["_CACHEDREMOTERELATION"]._serialized_end = 9955 - _globals["_SAMPLE"]._serialized_start = 9958 - _globals["_SAMPLE"]._serialized_end = 10231 - _globals["_RANGE"]._serialized_start = 10234 - _globals["_RANGE"]._serialized_end = 10379 - _globals["_SUBQUERYALIAS"]._serialized_start = 10381 - _globals["_SUBQUERYALIAS"]._serialized_end = 10495 - _globals["_REPARTITION"]._serialized_start = 10498 - _globals["_REPARTITION"]._serialized_end = 10640 - _globals["_SHOWSTRING"]._serialized_start = 10643 - _globals["_SHOWSTRING"]._serialized_end = 10785 - _globals["_HTMLSTRING"]._serialized_start = 10787 - _globals["_HTMLSTRING"]._serialized_end = 10901 - _globals["_STATSUMMARY"]._serialized_start = 10903 - _globals["_STATSUMMARY"]._serialized_end = 10995 - _globals["_STATDESCRIBE"]._serialized_start = 10997 - _globals["_STATDESCRIBE"]._serialized_end = 11078 - _globals["_STATCROSSTAB"]._serialized_start = 11080 - _globals["_STATCROSSTAB"]._serialized_end = 11181 - _globals["_STATCOV"]._serialized_start = 11183 - _globals["_STATCOV"]._serialized_end = 11279 - _globals["_STATCORR"]._serialized_start = 11282 - _globals["_STATCORR"]._serialized_end = 11419 - _globals["_STATAPPROXQUANTILE"]._serialized_start = 11422 - _globals["_STATAPPROXQUANTILE"]._serialized_end = 11586 - _globals["_STATFREQITEMS"]._serialized_start = 11588 - _globals["_STATFREQITEMS"]._serialized_end = 11713 - _globals["_STATSAMPLEBY"]._serialized_start = 11716 - _globals["_STATSAMPLEBY"]._serialized_end = 12025 - _globals["_STATSAMPLEBY_FRACTION"]._serialized_start = 11917 - _globals["_STATSAMPLEBY_FRACTION"]._serialized_end = 12016 - _globals["_NAFILL"]._serialized_start = 12028 - _globals["_NAFILL"]._serialized_end = 12162 - _globals["_NADROP"]._serialized_start = 12165 - _globals["_NADROP"]._serialized_end = 12299 - _globals["_NAREPLACE"]._serialized_start = 12302 - _globals["_NAREPLACE"]._serialized_end = 12598 - _globals["_NAREPLACE_REPLACEMENT"]._serialized_start = 12457 - _globals["_NAREPLACE_REPLACEMENT"]._serialized_end = 12598 - _globals["_TODF"]._serialized_start = 12600 - _globals["_TODF"]._serialized_end = 12688 - _globals["_WITHCOLUMNSRENAMED"]._serialized_start = 12691 - _globals["_WITHCOLUMNSRENAMED"]._serialized_end = 13073 - _globals["_WITHCOLUMNSRENAMED_RENAMECOLUMNSMAPENTRY"]._serialized_start = 12935 - _globals["_WITHCOLUMNSRENAMED_RENAMECOLUMNSMAPENTRY"]._serialized_end = 13002 - _globals["_WITHCOLUMNSRENAMED_RENAME"]._serialized_start = 13004 - _globals["_WITHCOLUMNSRENAMED_RENAME"]._serialized_end = 13073 - _globals["_WITHCOLUMNS"]._serialized_start = 13075 - _globals["_WITHCOLUMNS"]._serialized_end = 13194 - _globals["_WITHWATERMARK"]._serialized_start = 13197 - _globals["_WITHWATERMARK"]._serialized_end = 13331 - _globals["_HINT"]._serialized_start = 13334 - _globals["_HINT"]._serialized_end = 13466 - _globals["_UNPIVOT"]._serialized_start = 13469 - _globals["_UNPIVOT"]._serialized_end = 13796 - _globals["_UNPIVOT_VALUES"]._serialized_start = 13726 - _globals["_UNPIVOT_VALUES"]._serialized_end = 13785 - _globals["_TRANSPOSE"]._serialized_start = 13798 - _globals["_TRANSPOSE"]._serialized_end = 13920 - _globals["_UNRESOLVEDTABLEVALUEDFUNCTION"]._serialized_start = 13922 - _globals["_UNRESOLVEDTABLEVALUEDFUNCTION"]._serialized_end = 14047 - _globals["_TOSCHEMA"]._serialized_start = 14049 - _globals["_TOSCHEMA"]._serialized_end = 14155 - _globals["_REPARTITIONBYEXPRESSION"]._serialized_start = 14158 - _globals["_REPARTITIONBYEXPRESSION"]._serialized_end = 14361 - _globals["_MAPPARTITIONS"]._serialized_start = 14364 - _globals["_MAPPARTITIONS"]._serialized_end = 14596 - _globals["_GROUPMAP"]._serialized_start = 14599 - _globals["_GROUPMAP"]._serialized_end = 15449 - _globals["_TRANSFORMWITHSTATEINFO"]._serialized_start = 15452 - _globals["_TRANSFORMWITHSTATEINFO"]._serialized_end = 15675 - _globals["_COGROUPMAP"]._serialized_start = 15678 - _globals["_COGROUPMAP"]._serialized_end = 16204 - _globals["_APPLYINPANDASWITHSTATE"]._serialized_start = 16207 - _globals["_APPLYINPANDASWITHSTATE"]._serialized_end = 16564 - _globals["_COMMONINLINEUSERDEFINEDTABLEFUNCTION"]._serialized_start = 16567 - _globals["_COMMONINLINEUSERDEFINEDTABLEFUNCTION"]._serialized_end = 16811 - _globals["_PYTHONUDTF"]._serialized_start = 16814 - _globals["_PYTHONUDTF"]._serialized_end = 16991 - _globals["_COMMONINLINEUSERDEFINEDDATASOURCE"]._serialized_start = 16994 - _globals["_COMMONINLINEUSERDEFINEDDATASOURCE"]._serialized_end = 17145 - _globals["_PYTHONDATASOURCE"]._serialized_start = 17147 - _globals["_PYTHONDATASOURCE"]._serialized_end = 17222 - _globals["_COLLECTMETRICS"]._serialized_start = 17225 - _globals["_COLLECTMETRICS"]._serialized_end = 17361 - _globals["_PARSE"]._serialized_start = 17364 - _globals["_PARSE"]._serialized_end = 17774 - _globals["_PARSE_OPTIONSENTRY"]._serialized_start = 6045 - _globals["_PARSE_OPTIONSENTRY"]._serialized_end = 6103 - _globals["_PARSE_PARSEFORMAT"]._serialized_start = 17653 - _globals["_PARSE_PARSEFORMAT"]._serialized_end = 17763 - _globals["_ASOFJOIN"]._serialized_start = 17777 - _globals["_ASOFJOIN"]._serialized_end = 18252 - _globals["_LATERALJOIN"]._serialized_start = 18255 - _globals["_LATERALJOIN"]._serialized_end = 18485 + _globals["_RELATION"]._serialized_end = 4225 + _globals["_MLRELATION"]._serialized_start = 4228 + _globals["_MLRELATION"]._serialized_end = 4712 + _globals["_MLRELATION_TRANSFORM"]._serialized_start = 4440 + _globals["_MLRELATION_TRANSFORM"]._serialized_end = 4675 + _globals["_FETCH"]._serialized_start = 4715 + _globals["_FETCH"]._serialized_end = 5046 + _globals["_FETCH_METHOD"]._serialized_start = 4831 + _globals["_FETCH_METHOD"]._serialized_end = 5046 + _globals["_FETCH_METHOD_ARGS"]._serialized_start = 4919 + _globals["_FETCH_METHOD_ARGS"]._serialized_end = 5046 + _globals["_UNKNOWN"]._serialized_start = 5048 + _globals["_UNKNOWN"]._serialized_end = 5057 + _globals["_RELATIONCOMMON"]._serialized_start = 5060 + _globals["_RELATIONCOMMON"]._serialized_end = 5202 + _globals["_SQL"]._serialized_start = 5205 + _globals["_SQL"]._serialized_end = 5683 + _globals["_SQL_ARGSENTRY"]._serialized_start = 5499 + _globals["_SQL_ARGSENTRY"]._serialized_end = 5589 + _globals["_SQL_NAMEDARGUMENTSENTRY"]._serialized_start = 5591 + _globals["_SQL_NAMEDARGUMENTSENTRY"]._serialized_end = 5683 + _globals["_WITHRELATIONS"]._serialized_start = 5685 + _globals["_WITHRELATIONS"]._serialized_end = 5802 + _globals["_READ"]._serialized_start = 5805 + _globals["_READ"]._serialized_end = 6522 + _globals["_READ_NAMEDTABLE"]._serialized_start = 5983 + _globals["_READ_NAMEDTABLE"]._serialized_end = 6175 + _globals["_READ_NAMEDTABLE_OPTIONSENTRY"]._serialized_start = 6117 + _globals["_READ_NAMEDTABLE_OPTIONSENTRY"]._serialized_end = 6175 + _globals["_READ_DATASOURCE"]._serialized_start = 6178 + _globals["_READ_DATASOURCE"]._serialized_end = 6509 + _globals["_READ_DATASOURCE_OPTIONSENTRY"]._serialized_start = 6117 + _globals["_READ_DATASOURCE_OPTIONSENTRY"]._serialized_end = 6175 + _globals["_RELATIONCHANGES"]._serialized_start = 6525 + _globals["_RELATIONCHANGES"]._serialized_end = 6757 + _globals["_RELATIONCHANGES_OPTIONSENTRY"]._serialized_start = 6117 + _globals["_RELATIONCHANGES_OPTIONSENTRY"]._serialized_end = 6175 + _globals["_PROJECT"]._serialized_start = 6759 + _globals["_PROJECT"]._serialized_end = 6876 + _globals["_FILTER"]._serialized_start = 6878 + _globals["_FILTER"]._serialized_end = 6990 + _globals["_JOIN"]._serialized_start = 6993 + _globals["_JOIN"]._serialized_end = 7654 + _globals["_JOIN_JOINDATATYPE"]._serialized_start = 7332 + _globals["_JOIN_JOINDATATYPE"]._serialized_end = 7424 + _globals["_JOIN_JOINTYPE"]._serialized_start = 7427 + _globals["_JOIN_JOINTYPE"]._serialized_end = 7635 + _globals["_SETOPERATION"]._serialized_start = 7657 + _globals["_SETOPERATION"]._serialized_end = 8136 + _globals["_SETOPERATION_SETOPTYPE"]._serialized_start = 7973 + _globals["_SETOPERATION_SETOPTYPE"]._serialized_end = 8087 + _globals["_LIMIT"]._serialized_start = 8138 + _globals["_LIMIT"]._serialized_end = 8214 + _globals["_OFFSET"]._serialized_start = 8216 + _globals["_OFFSET"]._serialized_end = 8295 + _globals["_TAIL"]._serialized_start = 8297 + _globals["_TAIL"]._serialized_end = 8372 + _globals["_AGGREGATE"]._serialized_start = 8375 + _globals["_AGGREGATE"]._serialized_end = 9141 + _globals["_AGGREGATE_PIVOT"]._serialized_start = 8790 + _globals["_AGGREGATE_PIVOT"]._serialized_end = 8901 + _globals["_AGGREGATE_GROUPINGSETS"]._serialized_start = 8903 + _globals["_AGGREGATE_GROUPINGSETS"]._serialized_end = 8979 + _globals["_AGGREGATE_GROUPTYPE"]._serialized_start = 8982 + _globals["_AGGREGATE_GROUPTYPE"]._serialized_end = 9141 + _globals["_SORT"]._serialized_start = 9144 + _globals["_SORT"]._serialized_end = 9304 + _globals["_DROP"]._serialized_start = 9307 + _globals["_DROP"]._serialized_end = 9448 + _globals["_DEDUPLICATE"]._serialized_start = 9451 + _globals["_DEDUPLICATE"]._serialized_end = 9691 + _globals["_LOCALRELATION"]._serialized_start = 9693 + _globals["_LOCALRELATION"]._serialized_end = 9782 + _globals["_CACHEDLOCALRELATION"]._serialized_start = 9784 + _globals["_CACHEDLOCALRELATION"]._serialized_end = 9856 + _globals["_CHUNKEDCACHEDLOCALRELATION"]._serialized_start = 9858 + _globals["_CHUNKEDCACHEDLOCALRELATION"]._serialized_end = 9970 + _globals["_CACHEDREMOTERELATION"]._serialized_start = 9972 + _globals["_CACHEDREMOTERELATION"]._serialized_end = 10027 + _globals["_SAMPLE"]._serialized_start = 10030 + _globals["_SAMPLE"]._serialized_end = 10303 + _globals["_RANGE"]._serialized_start = 10306 + _globals["_RANGE"]._serialized_end = 10451 + _globals["_SUBQUERYALIAS"]._serialized_start = 10453 + _globals["_SUBQUERYALIAS"]._serialized_end = 10567 + _globals["_REPARTITION"]._serialized_start = 10570 + _globals["_REPARTITION"]._serialized_end = 10712 + _globals["_SHOWSTRING"]._serialized_start = 10715 + _globals["_SHOWSTRING"]._serialized_end = 10857 + _globals["_HTMLSTRING"]._serialized_start = 10859 + _globals["_HTMLSTRING"]._serialized_end = 10973 + _globals["_STATSUMMARY"]._serialized_start = 10975 + _globals["_STATSUMMARY"]._serialized_end = 11067 + _globals["_STATDESCRIBE"]._serialized_start = 11069 + _globals["_STATDESCRIBE"]._serialized_end = 11150 + _globals["_STATCROSSTAB"]._serialized_start = 11152 + _globals["_STATCROSSTAB"]._serialized_end = 11253 + _globals["_STATCOV"]._serialized_start = 11255 + _globals["_STATCOV"]._serialized_end = 11351 + _globals["_STATCORR"]._serialized_start = 11354 + _globals["_STATCORR"]._serialized_end = 11491 + _globals["_STATAPPROXQUANTILE"]._serialized_start = 11494 + _globals["_STATAPPROXQUANTILE"]._serialized_end = 11658 + _globals["_STATFREQITEMS"]._serialized_start = 11660 + _globals["_STATFREQITEMS"]._serialized_end = 11785 + _globals["_STATSAMPLEBY"]._serialized_start = 11788 + _globals["_STATSAMPLEBY"]._serialized_end = 12097 + _globals["_STATSAMPLEBY_FRACTION"]._serialized_start = 11989 + _globals["_STATSAMPLEBY_FRACTION"]._serialized_end = 12088 + _globals["_NAFILL"]._serialized_start = 12100 + _globals["_NAFILL"]._serialized_end = 12234 + _globals["_NADROP"]._serialized_start = 12237 + _globals["_NADROP"]._serialized_end = 12371 + _globals["_NAREPLACE"]._serialized_start = 12374 + _globals["_NAREPLACE"]._serialized_end = 12670 + _globals["_NAREPLACE_REPLACEMENT"]._serialized_start = 12529 + _globals["_NAREPLACE_REPLACEMENT"]._serialized_end = 12670 + _globals["_TODF"]._serialized_start = 12672 + _globals["_TODF"]._serialized_end = 12760 + _globals["_WITHCOLUMNSRENAMED"]._serialized_start = 12763 + _globals["_WITHCOLUMNSRENAMED"]._serialized_end = 13145 + _globals["_WITHCOLUMNSRENAMED_RENAMECOLUMNSMAPENTRY"]._serialized_start = 13007 + _globals["_WITHCOLUMNSRENAMED_RENAMECOLUMNSMAPENTRY"]._serialized_end = 13074 + _globals["_WITHCOLUMNSRENAMED_RENAME"]._serialized_start = 13076 + _globals["_WITHCOLUMNSRENAMED_RENAME"]._serialized_end = 13145 + _globals["_WITHCOLUMNS"]._serialized_start = 13147 + _globals["_WITHCOLUMNS"]._serialized_end = 13266 + _globals["_WITHWATERMARK"]._serialized_start = 13269 + _globals["_WITHWATERMARK"]._serialized_end = 13403 + _globals["_HINT"]._serialized_start = 13406 + _globals["_HINT"]._serialized_end = 13538 + _globals["_UNPIVOT"]._serialized_start = 13541 + _globals["_UNPIVOT"]._serialized_end = 13868 + _globals["_UNPIVOT_VALUES"]._serialized_start = 13798 + _globals["_UNPIVOT_VALUES"]._serialized_end = 13857 + _globals["_TRANSPOSE"]._serialized_start = 13870 + _globals["_TRANSPOSE"]._serialized_end = 13992 + _globals["_UNRESOLVEDTABLEVALUEDFUNCTION"]._serialized_start = 13994 + _globals["_UNRESOLVEDTABLEVALUEDFUNCTION"]._serialized_end = 14119 + _globals["_TOSCHEMA"]._serialized_start = 14121 + _globals["_TOSCHEMA"]._serialized_end = 14227 + _globals["_REPARTITIONBYEXPRESSION"]._serialized_start = 14230 + _globals["_REPARTITIONBYEXPRESSION"]._serialized_end = 14433 + _globals["_MAPPARTITIONS"]._serialized_start = 14436 + _globals["_MAPPARTITIONS"]._serialized_end = 14668 + _globals["_GROUPMAP"]._serialized_start = 14671 + _globals["_GROUPMAP"]._serialized_end = 15521 + _globals["_TRANSFORMWITHSTATEINFO"]._serialized_start = 15524 + _globals["_TRANSFORMWITHSTATEINFO"]._serialized_end = 15747 + _globals["_COGROUPMAP"]._serialized_start = 15750 + _globals["_COGROUPMAP"]._serialized_end = 16276 + _globals["_APPLYINPANDASWITHSTATE"]._serialized_start = 16279 + _globals["_APPLYINPANDASWITHSTATE"]._serialized_end = 16636 + _globals["_COMMONINLINEUSERDEFINEDTABLEFUNCTION"]._serialized_start = 16639 + _globals["_COMMONINLINEUSERDEFINEDTABLEFUNCTION"]._serialized_end = 16883 + _globals["_PYTHONUDTF"]._serialized_start = 16886 + _globals["_PYTHONUDTF"]._serialized_end = 17063 + _globals["_COMMONINLINEUSERDEFINEDDATASOURCE"]._serialized_start = 17066 + _globals["_COMMONINLINEUSERDEFINEDDATASOURCE"]._serialized_end = 17217 + _globals["_PYTHONDATASOURCE"]._serialized_start = 17219 + _globals["_PYTHONDATASOURCE"]._serialized_end = 17294 + _globals["_COLLECTMETRICS"]._serialized_start = 17297 + _globals["_COLLECTMETRICS"]._serialized_end = 17433 + _globals["_PARSE"]._serialized_start = 17436 + _globals["_PARSE"]._serialized_end = 17846 + _globals["_PARSE_OPTIONSENTRY"]._serialized_start = 6117 + _globals["_PARSE_OPTIONSENTRY"]._serialized_end = 6175 + _globals["_PARSE_PARSEFORMAT"]._serialized_start = 17725 + _globals["_PARSE_PARSEFORMAT"]._serialized_end = 17835 + _globals["_ASOFJOIN"]._serialized_start = 17849 + _globals["_ASOFJOIN"]._serialized_end = 18324 + _globals["_LATERALJOIN"]._serialized_start = 18327 + _globals["_LATERALJOIN"]._serialized_end = 18557 + _globals["_NEARESTBYJOIN"]._serialized_start = 18560 + _globals["_NEARESTBYJOIN"]._serialized_end = 18853 # @@protoc_insertion_point(module_scope) diff --git a/python/pyspark/sql/connect/proto/relations_pb2.pyi b/python/pyspark/sql/connect/proto/relations_pb2.pyi index 7b3968545ce0d..c99de778db4cd 100644 --- a/python/pyspark/sql/connect/proto/relations_pb2.pyi +++ b/python/pyspark/sql/connect/proto/relations_pb2.pyi @@ -111,6 +111,7 @@ class Relation(google.protobuf.message.Message): LATERAL_JOIN_FIELD_NUMBER: builtins.int CHUNKED_CACHED_LOCAL_RELATION_FIELD_NUMBER: builtins.int RELATION_CHANGES_FIELD_NUMBER: builtins.int + NEAREST_BY_JOIN_FIELD_NUMBER: builtins.int FILL_NA_FIELD_NUMBER: builtins.int DROP_NA_FIELD_NUMBER: builtins.int REPLACE_FIELD_NUMBER: builtins.int @@ -223,6 +224,8 @@ class Relation(google.protobuf.message.Message): @property def relation_changes(self) -> global___RelationChanges: ... @property + def nearest_by_join(self) -> global___NearestByJoin: ... + @property def fill_na(self) -> global___NAFill: """NA functions""" @property @@ -310,6 +313,7 @@ class Relation(google.protobuf.message.Message): lateral_join: global___LateralJoin | None = ..., chunked_cached_local_relation: global___ChunkedCachedLocalRelation | None = ..., relation_changes: global___RelationChanges | None = ..., + nearest_by_join: global___NearestByJoin | None = ..., fill_na: global___NAFill | None = ..., drop_na: global___NADrop | None = ..., replace: global___NAReplace | None = ..., @@ -395,6 +399,8 @@ class Relation(google.protobuf.message.Message): b"map_partitions", "ml_relation", b"ml_relation", + "nearest_by_join", + b"nearest_by_join", "offset", b"offset", "parse", @@ -524,6 +530,8 @@ class Relation(google.protobuf.message.Message): b"map_partitions", "ml_relation", b"ml_relation", + "nearest_by_join", + b"nearest_by_join", "offset", b"offset", "parse", @@ -633,6 +641,7 @@ class Relation(google.protobuf.message.Message): "lateral_join", "chunked_cached_local_relation", "relation_changes", + "nearest_by_join", "fill_na", "drop_na", "replace", @@ -4657,3 +4666,79 @@ class LateralJoin(google.protobuf.message.Message): ) -> None: ... global___LateralJoin = LateralJoin + +class NearestByJoin(google.protobuf.message.Message): + """Relation of type [[NearestByJoin]]. + + For each row on the left side, returns up to `num_results` rows from the right side ranked + by `ranking_expression`. + """ + + DESCRIPTOR: google.protobuf.descriptor.Descriptor + + LEFT_FIELD_NUMBER: builtins.int + RIGHT_FIELD_NUMBER: builtins.int + RANKING_EXPRESSION_FIELD_NUMBER: builtins.int + NUM_RESULTS_FIELD_NUMBER: builtins.int + JOIN_TYPE_FIELD_NUMBER: builtins.int + MODE_FIELD_NUMBER: builtins.int + DIRECTION_FIELD_NUMBER: builtins.int + @property + def left(self) -> global___Relation: + """(Required) Left (query) input relation.""" + @property + def right(self) -> global___Relation: + """(Required) Right (base) input relation.""" + @property + def ranking_expression(self) -> pyspark.sql.connect.proto.expressions_pb2.Expression: + """(Required) Scalar expression used to rank candidate rows on the right side.""" + num_results: builtins.int + """(Required) Maximum number of matches per left row. Must be between 1 and 100000.""" + join_type: builtins.str + """The following three fields use `string` (not typed enums) for parity with `AsOfJoin`, + which models analogous fields the same way. Validation happens server-side at planning time. + + (Required) The join type. Must be one of: "inner", "leftouter". + """ + mode: builtins.str + """(Required) Search algorithm contract. Must be one of: "approx", "exact".""" + direction: builtins.str + """(Required) Ranking direction. Must be one of: "distance", "similarity".""" + def __init__( + self, + *, + left: global___Relation | None = ..., + right: global___Relation | None = ..., + ranking_expression: pyspark.sql.connect.proto.expressions_pb2.Expression | None = ..., + num_results: builtins.int = ..., + join_type: builtins.str = ..., + mode: builtins.str = ..., + direction: builtins.str = ..., + ) -> None: ... + def HasField( + self, + field_name: typing_extensions.Literal[ + "left", b"left", "ranking_expression", b"ranking_expression", "right", b"right" + ], + ) -> builtins.bool: ... + def ClearField( + self, + field_name: typing_extensions.Literal[ + "direction", + b"direction", + "join_type", + b"join_type", + "left", + b"left", + "mode", + b"mode", + "num_results", + b"num_results", + "ranking_expression", + b"ranking_expression", + "right", + b"right", + ], + ) -> None: ... + +global___NearestByJoin = NearestByJoin diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py index b2586a2d7a18c..734b8cad62110 100644 --- a/python/pyspark/sql/dataframe.py +++ b/python/pyspark/sql/dataframe.py @@ -2865,6 +2865,73 @@ def lateralJoin( """ ... + def nearestByJoin( + self, + other: "DataFrame", + rankingExpression: Column, + numResults: int, + mode: str, + direction: str, + *, + joinType: str = "inner", + ) -> "DataFrame": + """ + Nearest-by top-K ranking join with another :class:`DataFrame`. For each row on the + left (query side), returns up to ``numResults`` rows from ``other`` (base side), ranked + by ``rankingExpression``. + + The current implementation evaluates the full cross-product of left and right and + bounds memory per left row by ``numResults``. Index-backed approximate strategies + (transparent to ``approx`` mode) are planned for a future release; until then, + pre-filter ``other`` when it is large. Tie-breaking among rows with equal ranking + values is unspecified. + + .. versionadded:: 4.2.0 + + Parameters + ---------- + other : :class:`DataFrame` + Right (base side) of the join - the candidate pool searched for each row of this + DataFrame. + rankingExpression : :class:`Column` + Scalar expression used to rank candidate rows on the right side. + numResults : int + Maximum number of matches per query row. Must be between 1 and 100000. + mode : str + Search algorithm contract. Must be one of: ``approx``, ``exact``. ``approx`` allows + the optimizer to use indexed or other approximate strategies when available; + ``exact`` forces brute-force evaluation and requires the ranking expression to be + deterministic. + direction : str + ``"distance"`` (smallest value first) or ``"similarity"`` (largest value first). + joinType : str, keyword-only, optional + Default ``inner``. Must be one of: ``inner``, ``leftouter``. + + Returns + ------- + :class:`DataFrame` + Joined DataFrame. + + Examples + -------- + >>> from pyspark.sql import functions as sf + >>> users = spark.createDataFrame( + ... [(1, 10.0), (2, 20.0), (3, 30.0)], ["user_id", "score"]) + >>> products = spark.createDataFrame( + ... [("A", 11.0), ("B", 22.0), ("C", 5.0)], ["product", "pscore"]) + >>> users.nearestByJoin( + ... products, -sf.abs(users.score - products.pscore), 1, "exact", "similarity" + ... ).select("user_id", "product").orderBy("user_id").show() + +-------+-------+ + |user_id|product| + +-------+-------+ + | 1| A| + | 2| B| + | 3| B| + +-------+-------+ + """ + ... + # TODO(SPARK-22947): Fix the DataFrame API. @dispatch_df_method def _joinAsOf( diff --git a/python/pyspark/sql/tests/connect/test_parity_nearest_by_join.py b/python/pyspark/sql/tests/connect/test_parity_nearest_by_join.py new file mode 100644 index 0000000000000..1fb0f5b620463 --- /dev/null +++ b/python/pyspark/sql/tests/connect/test_parity_nearest_by_join.py @@ -0,0 +1,30 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +from pyspark.sql.tests.test_nearest_by_join import NearestByJoinTestsMixin +from pyspark.testing.connectutils import ReusedConnectTestCase + + +class NearestByJoinParityTests(NearestByJoinTestsMixin, ReusedConnectTestCase): + pass + + +if __name__ == "__main__": + from pyspark.testing import main + + main() diff --git a/python/pyspark/sql/tests/test_nearest_by_join.py b/python/pyspark/sql/tests/test_nearest_by_join.py new file mode 100644 index 0000000000000..fdee3043289ef --- /dev/null +++ b/python/pyspark/sql/tests/test_nearest_by_join.py @@ -0,0 +1,270 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +from pyspark.errors import AnalysisException +from pyspark.sql import Row +from pyspark.sql import functions as sf +from pyspark.testing import assertDataFrameEqual +from pyspark.testing.sqlutils import ReusedSQLTestCase + + +class NearestByJoinTestsMixin: + """Mixin run against both classic (`ReusedSQLTestCase`) and Connect + (`ReusedConnectTestCase`) to ensure parity between the two paths.""" + + @property + def users(self): + return self.spark.createDataFrame([(1, 10.0), (2, 20.0), (3, 30.0)], ["user_id", "score"]) + + @property + def products(self): + return self.spark.createDataFrame( + [("A", 11.0), ("B", 22.0), ("C", 5.0)], ["product", "pscore"] + ) + + def test_inner_similarity_k1(self): + users, products = self.users, self.products + result = ( + users.nearestByJoin( + products, + -sf.abs(users.score - products.pscore), + numResults=1, + mode="approx", + direction="similarity", + ) + .select("user_id", "product") + .orderBy("user_id") + ) + assertDataFrameEqual( + result, + [Row(user_id=1, product="A"), Row(user_id=2, product="B"), Row(user_id=3, product="B")], + ) + + def test_inner_distance_k2(self): + users, products = self.users, self.products + result = ( + users.nearestByJoin( + products, + sf.abs(users.score - products.pscore), + numResults=2, + mode="approx", + direction="distance", + ) + .select("user_id", "product") + .orderBy("user_id", "product") + ) + assertDataFrameEqual( + result, + [ + Row(user_id=1, product="A"), + Row(user_id=1, product="C"), + Row(user_id=2, product="A"), + Row(user_id=2, product="B"), + Row(user_id=3, product="A"), + Row(user_id=3, product="B"), + ], + ) + + def test_left_outer_with_empty_right(self): + users, products = self.users, self.products + empty = products.filter(sf.lit(False)) + result = ( + users.nearestByJoin( + empty, + -sf.abs(users.score - empty.pscore), + numResults=1, + mode="exact", + direction="similarity", + joinType="leftouter", + ) + .select("user_id", "product") + .orderBy("user_id") + ) + assertDataFrameEqual( + result, + [ + Row(user_id=1, product=None), + Row(user_id=2, product=None), + Row(user_id=3, product=None), + ], + ) + + def test_select_star_schema_has_no_internal_columns(self): + users, products = self.users, self.products + result = users.nearestByJoin( + products, + -sf.abs(users.score - products.pscore), + numResults=1, + mode="exact", + direction="similarity", + ) + # No `__qid`, `__nearest_matches__`, or other rewrite-internal columns leak through. + assert sorted(result.columns) == ["product", "pscore", "score", "user_id"] + + def test_invalid_num_results_low(self): + users, products = self.users, self.products + with self.assertRaises(AnalysisException) as pe: + users.nearestByJoin( + products, + -sf.abs(users.score - products.pscore), + numResults=0, + mode="approx", + direction="similarity", + ) + self.check_error( + exception=pe.exception, + errorClass="NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE", + messageParameters={"numResults": "0", "min": "1", "max": "100000"}, + ) + + def test_invalid_num_results_high(self): + users, products = self.users, self.products + with self.assertRaises(AnalysisException) as pe: + users.nearestByJoin( + products, + -sf.abs(users.score - products.pscore), + numResults=200000, + mode="approx", + direction="similarity", + ) + self.check_error( + exception=pe.exception, + errorClass="NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE", + messageParameters={"numResults": "200000", "min": "1", "max": "100000"}, + ) + + def test_invalid_join_type(self): + users, products = self.users, self.products + with self.assertRaises(AnalysisException) as pe: + users.nearestByJoin( + products, + -sf.abs(users.score - products.pscore), + numResults=1, + mode="approx", + direction="similarity", + joinType="outer", + ) + self.check_error( + exception=pe.exception, + errorClass="NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE", + messageParameters={"joinType": "outer", "supported": "'INNER', 'LEFT OUTER'"}, + ) + + def test_invalid_mode(self): + users, products = self.users, self.products + with self.assertRaises(AnalysisException) as pe: + users.nearestByJoin( + products, + -sf.abs(users.score - products.pscore), + numResults=1, + mode="bogus", + direction="similarity", + ) + self.check_error( + exception=pe.exception, + errorClass="NEAREST_BY_JOIN.UNSUPPORTED_MODE", + messageParameters={"mode": "bogus", "supported": "'approx', 'exact'"}, + ) + + def test_invalid_direction(self): + users, products = self.users, self.products + with self.assertRaises(AnalysisException) as pe: + users.nearestByJoin( + products, + -sf.abs(users.score - products.pscore), + numResults=1, + mode="approx", + direction="elsewhere", + ) + self.check_error( + exception=pe.exception, + errorClass="NEAREST_BY_JOIN.UNSUPPORTED_DIRECTION", + messageParameters={ + "direction": "elsewhere", + "supported": "'distance', 'similarity'", + }, + ) + + def test_rejected_when_crossjoin_disabled(self): + users, products = self.users, self.products + with self.sql_conf({"spark.sql.crossJoin.enabled": "false"}): + with self.assertRaises(AnalysisException) as pe: + users.nearestByJoin( + products, + -sf.abs(users.score - products.pscore), + numResults=1, + mode="exact", + direction="similarity", + ).collect() + self.check_error( + exception=pe.exception, + errorClass="NEAREST_BY_JOIN.CROSS_JOIN_NOT_ENABLED", + messageParameters={}, + ) + + def test_exact_with_nondeterministic_ranking_rejected(self): + users, products = self.users, self.products + # Use an explicit seed (`rand(0)`) so the rendered expression in the error message is + # byte-stable. Without it, Spark assigns a random seed at analysis and the message + # parameter becomes `"(rand() + pscore)"`, which can't be asserted on. + with self.assertRaises(AnalysisException) as pe: + users.nearestByJoin( + products, + sf.rand(0) + products.pscore, + numResults=1, + mode="exact", + direction="similarity", + ).collect() + self.check_error( + exception=pe.exception, + errorClass="NEAREST_BY_JOIN.EXACT_WITH_NONDETERMINISTIC_EXPRESSION", + messageParameters={"expression": '"(rand(0) + pscore)"'}, + ) + + def test_streaming_inputs_rejected(self): + streaming_users = ( + self.spark.readStream.format("rate") + .option("rowsPerSecond", 1) + .load() + .selectExpr("CAST(value AS INT) AS user_id", "CAST(value AS DOUBLE) AS score") + ) + products = self.products + with self.assertRaises(AnalysisException) as pe: + # `.schema` forces analysis without starting the streaming query. + _ = streaming_users.nearestByJoin( + products, + -sf.abs(streaming_users.score - products.pscore), + numResults=1, + mode="exact", + direction="similarity", + ).schema + self.check_error( + exception=pe.exception, + errorClass="NEAREST_BY_JOIN.STREAMING_NOT_SUPPORTED", + messageParameters={}, + ) + + +class NearestByJoinTests(NearestByJoinTestsMixin, ReusedSQLTestCase): + pass + + +if __name__ == "__main__": + from pyspark.testing import main + + main() diff --git a/sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala b/sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala index c3c983c17bb02..38765262e1fc5 100644 --- a/sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala +++ b/sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala @@ -912,6 +912,76 @@ abstract class Dataset[T] extends Serializable { */ def lateralJoin(right: Dataset[_], joinExprs: Column, joinType: String): DataFrame + /** + * Nearest-by top-K ranking join with another `DataFrame`, using the default `inner` join type. + * For each row on the left (query side), returns up to `numResults` rows from `right` (base + * side), ranked by `rankingExpression`. + * + * Equivalent SQL (with `mode = "exact"` and `direction = "similarity"`): + * {{{ + * left INNER JOIN right EXACT NEAREST numResults BY SIMILARITY rankingExpression + * }}} + * + * The current implementation evaluates the full cross-product of left and right and bounds + * memory per left row by `numResults`. Index-backed approximate strategies (transparent to + * `approx` mode) are planned for a future release; until then, pre-filter the right side when + * it is large. Tie-breaking among rows with equal ranking values is unspecified. + * + * @param right + * Right (base side) of the join - the candidate pool searched for each row of this Dataset. + * @param rankingExpression + * Scalar expression used to rank candidate rows. + * @param numResults + * Maximum number of matches per query row. Must be between 1 and 100000. + * @param mode + * Search algorithm contract. Must be one of: `approx`, `exact`. `approx` allows the optimizer + * to use indexed or other approximate strategies when available; `exact` forces brute-force + * evaluation and requires the ranking expression to be deterministic. + * @param direction + * `"distance"` (smallest value first) or `"similarity"` (largest value first). + * @group untypedrel + * @since 4.2.0 + */ + def nearestByJoin( + right: Dataset[_], + rankingExpression: Column, + numResults: Int, + mode: String, + direction: String): DataFrame + + /** + * Nearest-by top-K ranking join with another `DataFrame`. + * + * The current implementation evaluates the full cross-product of left and right and bounds + * memory per left row by `numResults`. Index-backed approximate strategies (transparent to + * `approx` mode) are planned for a future release; until then, pre-filter the right side when + * it is large. Tie-breaking among rows with equal ranking values is unspecified. + * + * @param right + * Right (base side) of the join - the candidate pool searched for each row of this Dataset. + * @param rankingExpression + * Scalar expression used to rank candidate rows. + * @param numResults + * Maximum number of matches per query row. Must be between 1 and 100000. + * @param mode + * Search algorithm contract. Must be one of: `approx`, `exact`. `approx` allows the optimizer + * to use indexed or other approximate strategies when available; `exact` forces brute-force + * evaluation and requires the ranking expression to be deterministic. + * @param direction + * `"distance"` (smallest value first) or `"similarity"` (largest value first). + * @param joinType + * Type of join to perform. Must be one of: `inner`, `leftouter`. + * @group untypedrel + * @since 4.2.0 + */ + def nearestByJoin( + right: Dataset[_], + rankingExpression: Column, + numResults: Int, + mode: String, + direction: String, + joinType: String): DataFrame + protected def sortInternal(global: Boolean, sortExprs: Seq[Column]): Dataset[T] /** diff --git a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/plans/NearestByJoinValidation.scala b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/plans/NearestByJoinValidation.scala new file mode 100644 index 0000000000000..8ebac8e73c671 --- /dev/null +++ b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/plans/NearestByJoinValidation.scala @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.catalyst.plans + +/** + * Acceptance lists for the `NEAREST BY` join API. + */ +private[sql] object NearestByJoinValidation { + + /** Upper bound on `numResults`. Mirrors the K-overload limit of `MaxMinByK`. */ + val MaxNumResults: Int = 100000 + + /** + * Strings accepted by `joinType` after lower-casing and stripping `_` (so e.g. `LEFT_OUTER` + * canonicalizes to `leftouter`). Every consumer must apply the same canonicalization before + * checking membership. + */ + val SupportedJoinTypes: Seq[String] = Seq("inner", "leftouter", "left") + + /** Display form for `supported` in `NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE` error messages. */ + val SupportedJoinTypeDisplay: String = "'INNER', 'LEFT OUTER'" + + /** Strings accepted by `mode`. Lower-cased before membership check. */ + val SupportedModes: Seq[String] = Seq("approx", "exact") + + /** Strings accepted by `direction`. Lower-cased before membership check. */ + val SupportedDirections: Seq[String] = Seq("distance", "similarity") +} diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala index 569cd05a46ba8..790307e44ec94 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala @@ -184,7 +184,8 @@ object LateralJoinType { object NearestByDirection { - val supported = Seq("distance", "similarity") + /** @see [[NearestByJoinValidation.SupportedDirections]] */ + val supported: Seq[String] = NearestByJoinValidation.SupportedDirections def apply(direction: String): NearestByDirection = { direction.toLowerCase(Locale.ROOT) match { @@ -207,13 +208,11 @@ case object NearestBySimilarity extends NearestByDirection object NearestByJoinType { - /** Strings accepted by the Dataset API. */ - val supported = Seq("inner", "leftouter", "left", "left_outer") + /** @see [[NearestByJoinValidation.SupportedJoinTypes]] */ + val supported: Seq[String] = NearestByJoinValidation.SupportedJoinTypes - /** Display string used in `NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE` error messages. Matches the - * parser-side wording so the same error class reports the same `supported` value across the - * SQL and DataFrame paths. */ - val supportedDisplay = "'INNER', 'LEFT OUTER'" + /** @see [[NearestByJoinValidation.SupportedJoinTypeDisplay]] */ + val supportedDisplay: String = NearestByJoinValidation.SupportedJoinTypeDisplay def apply(typ: String): JoinType = typ.toLowerCase(Locale.ROOT).replace("_", "") match { case "inner" => Inner @@ -229,7 +228,8 @@ object NearestByJoinType { object NearestByJoinMode { - val supported = Seq("approx", "exact") + /** @see [[NearestByJoinValidation.SupportedModes]] */ + val supported: Seq[String] = NearestByJoinValidation.SupportedModes /** Returns true for APPROX, false for EXACT. */ def apply(mode: String): Boolean = mode.toLowerCase(Locale.ROOT) match { diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/NearestByJoin.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/NearestByJoin.scala index 9df79ba128b8a..6a5c94d4a1df5 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/NearestByJoin.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/NearestByJoin.scala @@ -18,12 +18,12 @@ package org.apache.spark.sql.catalyst.plans.logical import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression} -import org.apache.spark.sql.catalyst.plans.{Inner, JoinType, LeftOuter, NearestByDirection} +import org.apache.spark.sql.catalyst.plans.{Inner, JoinType, LeftOuter, NearestByDirection, NearestByJoinValidation} import org.apache.spark.sql.catalyst.trees.TreePattern._ object NearestByJoin { - /** Upper bound on `numResults`. Mirrors the K-overload limit of `MaxMinByK`. */ - val MaxNumResults: Int = 100000 + /** @see [[NearestByJoinValidation.MaxNumResults]] */ + val MaxNumResults: Int = NearestByJoinValidation.MaxNumResults } /** diff --git a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/DataFrameNearestByJoinSuite.scala b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/DataFrameNearestByJoinSuite.scala new file mode 100644 index 0000000000000..00d7c4f80b09d --- /dev/null +++ b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/DataFrameNearestByJoinSuite.scala @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql + +import org.apache.spark.sql.connect.test.{QueryTest, RemoteSparkSession} +import org.apache.spark.sql.functions._ + +/** + * End-to-end Connect-side coverage for `Dataset.nearestByJoin`. Mirrors the + * `DataFrameNearestByJoinSuite` in `sql/core` for the classic path; this suite ensures the same + * API behaves correctly when invoked through the Connect client (proto serialization, server-side + * proto-to-catalyst translation in `SparkConnectPlanner.transformNearestByJoin`, and result + * roundtrip). + */ +class DataFrameNearestByJoinSuite extends QueryTest with RemoteSparkSession { + import testImplicits._ + + private lazy val users = Seq((1, 10.0), (2, 20.0), (3, 30.0)).toDF("user_id", "score") + + private lazy val products = Seq(("A", 11.0), ("B", 22.0), ("C", 5.0)).toDF("product", "pscore") + + test("inner approx similarity k=1") { + checkAnswer( + users + .nearestByJoin( + right = products, + rankingExpression = -abs(users("score") - products("pscore")), + numResults = 1, + mode = "approx", + direction = "similarity") + .select("user_id", "product") + .orderBy("user_id"), + Seq(Row(1, "A"), Row(2, "B"), Row(3, "B"))) + } + + test("inner approx distance k=2") { + checkAnswer( + users + .nearestByJoin( + right = products, + rankingExpression = abs(users("score") - products("pscore")), + numResults = 2, + mode = "approx", + direction = "distance") + .select("user_id", "product") + .orderBy("user_id", "product"), + Seq(Row(1, "A"), Row(1, "C"), Row(2, "A"), Row(2, "B"), Row(3, "A"), Row(3, "B"))) + } + + test("left outer with empty right preserves left rows with NULLs") { + val emptyProducts = products.filter(lit(false)) + checkAnswer( + users + .nearestByJoin( + right = emptyProducts, + rankingExpression = -abs(users("score") - emptyProducts("pscore")), + numResults = 1, + mode = "exact", + direction = "similarity", + joinType = "leftouter") + .select("user_id", "product") + .orderBy("user_id"), + Seq(Row(1, null), Row(2, null), Row(3, null))) + } + + test("output schema has no rewrite-internal columns") { + val result = users.nearestByJoin( + right = products, + rankingExpression = -abs(users("score") - products("pscore")), + numResults = 1, + mode = "exact", + direction = "similarity") + // Only the user-visible columns flow through; no `__qid`, `__nearest_matches__`, etc. + assert(result.columns.toSet === Set("user_id", "score", "product", "pscore")) + } + + test("invalid mode is rejected") { + val ex = intercept[AnalysisException] { + users.nearestByJoin( + right = products, + rankingExpression = -abs(users("score") - products("pscore")), + numResults = 1, + mode = "bogus", + direction = "similarity") + } + assert(ex.getCondition === "NEAREST_BY_JOIN.UNSUPPORTED_MODE") + } +} diff --git a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala index 16a2bf85de4ab..199736da92ac6 100644 --- a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala +++ b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala @@ -516,6 +516,29 @@ class PlanGenerationTestSuite extends ConnectFunSuite with Logging { left.crossJoin(right) } + test("nearestByJoin inner_approx_similarity") { + left + .as("l") + .nearestByJoin( + right = right.as("r"), + rankingExpression = fn.col("l.a") + fn.col("r.a"), + numResults = 1, + mode = "approx", + direction = "similarity") + } + + test("nearestByJoin leftouter_exact_distance") { + left + .as("l") + .nearestByJoin( + right = right.as("r"), + rankingExpression = fn.col("l.a") + fn.col("r.a"), + numResults = 5, + mode = "exact", + direction = "distance", + joinType = "leftouter") + } + test("sortWithinPartitions strings") { simple.sortWithinPartitions("a", "id") } diff --git a/sql/connect/common/src/main/protobuf/spark/connect/relations.proto b/sql/connect/common/src/main/protobuf/spark/connect/relations.proto index 57c4ed7be3c84..95cc9281d8cad 100644 --- a/sql/connect/common/src/main/protobuf/spark/connect/relations.proto +++ b/sql/connect/common/src/main/protobuf/spark/connect/relations.proto @@ -82,6 +82,7 @@ message Relation { LateralJoin lateral_join = 44; ChunkedCachedLocalRelation chunked_cached_local_relation = 45; RelationChanges relation_changes = 46; + NearestByJoin nearest_by_join = 47; // NA functions NAFill fill_na = 90; @@ -1276,3 +1277,33 @@ message LateralJoin { // (Required) The join type. Join.JoinType join_type = 4; } + +// Relation of type [[NearestByJoin]]. +// +// For each row on the left side, returns up to `num_results` rows from the right side ranked +// by `ranking_expression`. +message NearestByJoin { + // (Required) Left (query) input relation. + Relation left = 1; + + // (Required) Right (base) input relation. + Relation right = 2; + + // (Required) Scalar expression used to rank candidate rows on the right side. + Expression ranking_expression = 3; + + // (Required) Maximum number of matches per left row. Must be between 1 and 100000. + int32 num_results = 4; + + // The following three fields use `string` (not typed enums) for parity with `AsOfJoin`, + // which models analogous fields the same way. Validation happens server-side at planning time. + + // (Required) The join type. Must be one of: "inner", "leftouter". + string join_type = 5; + + // (Required) Search algorithm contract. Must be one of: "approx", "exact". + string mode = 6; + + // (Required) Ranking direction. Must be one of: "distance", "similarity". + string direction = 7; +} diff --git a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala index b57ea66bb1f7d..34c685213711c 100644 --- a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala +++ b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala @@ -36,6 +36,7 @@ import org.apache.spark.sql.catalyst.ScalaReflection import org.apache.spark.sql.catalyst.encoders.AgnosticEncoder import org.apache.spark.sql.catalyst.encoders.AgnosticEncoders._ import org.apache.spark.sql.catalyst.expressions.OrderUtils +import org.apache.spark.sql.catalyst.plans.NearestByJoinValidation import org.apache.spark.sql.connect.ColumnNodeToProtoConverter.{toExpr, toLiteral, toTypedExpr} import org.apache.spark.sql.connect.ConnectConversions._ import org.apache.spark.sql.connect.client.SparkResult @@ -421,6 +422,52 @@ class Dataset[T] private[sql] ( lateralJoin(right, Some(joinExprs), joinType) } + private def nearestByJoinImpl( + right: sql.Dataset[_], + rankingExpression: Column, + numResults: Int, + joinType: String, + mode: String, + direction: String): DataFrame = { + // Validate locally so Connect users see the same errors as the classic path without a + // server round-trip. The validation logic mirrors `NearestByJoinType.apply` / + // `NearestByJoinMode.apply` / `NearestByDirection.apply` in sql/catalyst, which + // `sql/connect/common` cannot import; the acceptance lists themselves are shared via + // `NearestByJoinValidation` in sql-api. + Dataset.validateNearestByJoinArgs(numResults, joinType, mode, direction) + sparkSession.newDataFrame(Seq(rankingExpression)) { builder => + builder.getNearestByJoinBuilder + .setLeft(plan.getRoot) + .setRight(right.plan.getRoot) + .setRankingExpression(toExpr(rankingExpression)) + .setNumResults(numResults) + .setJoinType(joinType) + .setMode(mode) + .setDirection(direction) + } + } + + /** @inheritdoc */ + def nearestByJoin( + right: sql.Dataset[_], + rankingExpression: Column, + numResults: Int, + mode: String, + direction: String): DataFrame = { + nearestByJoinImpl(right, rankingExpression, numResults, "inner", mode, direction) + } + + /** @inheritdoc */ + def nearestByJoin( + right: sql.Dataset[_], + rankingExpression: Column, + numResults: Int, + mode: String, + direction: String, + joinType: String): DataFrame = { + nearestByJoinImpl(right, rankingExpression, numResults, joinType, mode, direction) + } + override protected def sortInternal(global: Boolean, sortCols: Seq[Column]): Dataset[T] = { val sortExprs = sortCols.map { c => ColumnNodeToProtoConverter(c.sortOrder).getSortOrder @@ -1569,3 +1616,47 @@ class Dataset[T] private[sql] ( override def queryExecution: QueryExecution = throw ConnectClientUnsupportedErrors.queryExecution() } + +private[sql] object Dataset { + + private[connect] def validateNearestByJoinArgs( + numResults: Int, + joinType: String, + mode: String, + direction: String): Unit = { + if (numResults < 1 || numResults > NearestByJoinValidation.MaxNumResults) { + throw new AnalysisException( + errorClass = "NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE", + messageParameters = Map( + "numResults" -> numResults.toString, + "min" -> "1", + "max" -> NearestByJoinValidation.MaxNumResults.toString)) + } + val canonicalJoinType = joinType.toLowerCase(java.util.Locale.ROOT).replace("_", "") + if (!NearestByJoinValidation.SupportedJoinTypes.contains(canonicalJoinType)) { + throw new AnalysisException( + errorClass = "NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE", + messageParameters = Map( + "joinType" -> joinType, + "supported" -> NearestByJoinValidation.SupportedJoinTypeDisplay)) + } + if (!NearestByJoinValidation.SupportedModes.contains( + mode.toLowerCase(java.util.Locale.ROOT))) { + throw new AnalysisException( + errorClass = "NEAREST_BY_JOIN.UNSUPPORTED_MODE", + messageParameters = Map( + "mode" -> mode, + "supported" -> + NearestByJoinValidation.SupportedModes.mkString("'", "', '", "'"))) + } + if (!NearestByJoinValidation.SupportedDirections.contains( + direction.toLowerCase(java.util.Locale.ROOT))) { + throw new AnalysisException( + errorClass = "NEAREST_BY_JOIN.UNSUPPORTED_DIRECTION", + messageParameters = Map( + "direction" -> direction, + "supported" -> + NearestByJoinValidation.SupportedDirections.mkString("'", "', '", "'"))) + } + } +} diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/nearestByJoin_inner_approx_similarity.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/nearestByJoin_inner_approx_similarity.explain new file mode 100644 index 0000000000000..8e3750b4c4a76 --- /dev/null +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/nearestByJoin_inner_approx_similarity.explain @@ -0,0 +1,5 @@ +'NearestByJoin Inner, true, 1, (a#0 + a#0), NearestBySimilarity +:- SubqueryAlias l +: +- LocalRelation , [id#0L, a#0, b#0] ++- SubqueryAlias r + +- LocalRelation , [a#0, id#0L, payload#0] diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/nearestByJoin_leftouter_exact_distance.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/nearestByJoin_leftouter_exact_distance.explain new file mode 100644 index 0000000000000..67539c3964b1d --- /dev/null +++ b/sql/connect/common/src/test/resources/query-tests/explain-results/nearestByJoin_leftouter_exact_distance.explain @@ -0,0 +1,5 @@ +'NearestByJoin LeftOuter, false, 5, (a#0 + a#0), NearestByDistance +:- SubqueryAlias l +: +- LocalRelation , [id#0L, a#0, b#0] ++- SubqueryAlias r + +- LocalRelation , [a#0, id#0L, payload#0] diff --git a/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_inner_approx_similarity.json b/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_inner_approx_similarity.json new file mode 100644 index 0000000000000..ca4f2919e55c6 --- /dev/null +++ b/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_inner_approx_similarity.json @@ -0,0 +1,109 @@ +{ + "common": { + "planId": "4" + }, + "nearestByJoin": { + "left": { + "common": { + "planId": "1" + }, + "subqueryAlias": { + "input": { + "common": { + "planId": "0" + }, + "localRelation": { + "schema": "struct\u003cid:bigint,a:int,b:double\u003e" + } + }, + "alias": "l" + } + }, + "right": { + "common": { + "planId": "3" + }, + "subqueryAlias": { + "input": { + "common": { + "planId": "2" + }, + "localRelation": { + "schema": "struct\u003ca:int,id:bigint,payload:binary\u003e" + } + }, + "alias": "r" + } + }, + "rankingExpression": { + "unresolvedFunction": { + "functionName": "+", + "arguments": [{ + "unresolvedAttribute": { + "unparsedIdentifier": "l.a" + }, + "common": { + "origin": { + "jvmOrigin": { + "stackTrace": [{ + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.functions$", + "methodName": "col", + "fileName": "functions.scala" + }, { + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite", + "methodName": "~~trimmed~anonfun~~", + "fileName": "PlanGenerationTestSuite.scala" + }] + } + } + } + }, { + "unresolvedAttribute": { + "unparsedIdentifier": "r.a" + }, + "common": { + "origin": { + "jvmOrigin": { + "stackTrace": [{ + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.functions$", + "methodName": "col", + "fileName": "functions.scala" + }, { + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite", + "methodName": "~~trimmed~anonfun~~", + "fileName": "PlanGenerationTestSuite.scala" + }] + } + } + } + }], + "isInternal": false + }, + "common": { + "origin": { + "jvmOrigin": { + "stackTrace": [{ + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.Column", + "methodName": "$plus", + "fileName": "Column.scala" + }, { + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite", + "methodName": "~~trimmed~anonfun~~", + "fileName": "PlanGenerationTestSuite.scala" + }] + } + } + } + }, + "numResults": 1, + "joinType": "inner", + "mode": "approx", + "direction": "similarity" + } +} \ No newline at end of file diff --git a/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_inner_approx_similarity.proto.bin b/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_inner_approx_similarity.proto.bin new file mode 100644 index 0000000000000000000000000000000000000000..8dbeb994d8fcd0ec1c05fefea2857549b3e11cc2 GIT binary patch literal 708 zcmdUtF-rq66vzASdJaLb#H}KhQly+U!zlsf=jALtMc00nAe38MJB_blZ(Pc+tu}tgKj|>aOvJG+H&7QtH+oao^gSi+vykGsqX#& literal 0 HcmV?d00001 diff --git a/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_leftouter_exact_distance.json b/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_leftouter_exact_distance.json new file mode 100644 index 0000000000000..877bff8f90c8e --- /dev/null +++ b/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_leftouter_exact_distance.json @@ -0,0 +1,109 @@ +{ + "common": { + "planId": "4" + }, + "nearestByJoin": { + "left": { + "common": { + "planId": "1" + }, + "subqueryAlias": { + "input": { + "common": { + "planId": "0" + }, + "localRelation": { + "schema": "struct\u003cid:bigint,a:int,b:double\u003e" + } + }, + "alias": "l" + } + }, + "right": { + "common": { + "planId": "3" + }, + "subqueryAlias": { + "input": { + "common": { + "planId": "2" + }, + "localRelation": { + "schema": "struct\u003ca:int,id:bigint,payload:binary\u003e" + } + }, + "alias": "r" + } + }, + "rankingExpression": { + "unresolvedFunction": { + "functionName": "+", + "arguments": [{ + "unresolvedAttribute": { + "unparsedIdentifier": "l.a" + }, + "common": { + "origin": { + "jvmOrigin": { + "stackTrace": [{ + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.functions$", + "methodName": "col", + "fileName": "functions.scala" + }, { + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite", + "methodName": "~~trimmed~anonfun~~", + "fileName": "PlanGenerationTestSuite.scala" + }] + } + } + } + }, { + "unresolvedAttribute": { + "unparsedIdentifier": "r.a" + }, + "common": { + "origin": { + "jvmOrigin": { + "stackTrace": [{ + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.functions$", + "methodName": "col", + "fileName": "functions.scala" + }, { + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite", + "methodName": "~~trimmed~anonfun~~", + "fileName": "PlanGenerationTestSuite.scala" + }] + } + } + } + }], + "isInternal": false + }, + "common": { + "origin": { + "jvmOrigin": { + "stackTrace": [{ + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.Column", + "methodName": "$plus", + "fileName": "Column.scala" + }, { + "classLoaderName": "app", + "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite", + "methodName": "~~trimmed~anonfun~~", + "fileName": "PlanGenerationTestSuite.scala" + }] + } + } + } + }, + "numResults": 5, + "joinType": "leftouter", + "mode": "exact", + "direction": "distance" + } +} \ No newline at end of file diff --git a/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_leftouter_exact_distance.proto.bin b/sql/connect/common/src/test/resources/query-tests/queries/nearestByJoin_leftouter_exact_distance.proto.bin new file mode 100644 index 0000000000000000000000000000000000000000..a671071c556ed6049568d9de951d447d52b2fa1f GIT binary patch literal 709 zcmdUrze)o^5XQ52InYHgtX)J5AxJpm7N;N_!6Fv6f>w5uy-i%)?4J7vOmjsZA&(%4 z60o=N#kBFl1#Mz0*bFnn_xon}aL}lK1n&_(!9fGf&=KyIP3@*lTGkoQTwi4AP>h9g zJxG^zm}c4!B|jrblC%_uGjxn;*#tLTagv$yb@LBj@7Ct%1>q+| y9?{^m%4>IRUgui+iHf66F1<~*s_~D5Zbk8?tEAK%7 literal 0 HcmV?d00001 diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala index 12d0c1ce12a43..dff80cb242687 100644 --- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala +++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala @@ -159,6 +159,8 @@ class SparkConnectPlanner( case proto.Relation.RelTypeCase.JOIN => transformJoinOrJoinWith(rel.getJoin) case proto.Relation.RelTypeCase.AS_OF_JOIN => transformAsOfJoin(rel.getAsOfJoin) case proto.Relation.RelTypeCase.LATERAL_JOIN => transformLateralJoin(rel.getLateralJoin) + case proto.Relation.RelTypeCase.NEAREST_BY_JOIN => + transformNearestByJoin(rel.getNearestByJoin) case proto.Relation.RelTypeCase.DEDUPLICATE => transformDeduplicate(rel.getDeduplicate) case proto.Relation.RelTypeCase.SET_OP => transformSetOperation(rel.getSetOp) case proto.Relation.RelTypeCase.SORT => transformSort(rel.getSort) @@ -2567,6 +2569,28 @@ class SparkConnectPlanner( condition = joinCondition) } + private def transformNearestByJoin(rel: proto.NearestByJoin): LogicalPlan = { + assertPlan(rel.hasLeft && rel.hasRight, "Both join sides must be present") + assertPlan(rel.hasRankingExpression, "Ranking expression must be present") + // proto3 string fields default to "" when not set; reject the empty case explicitly so the + // user sees a "must be set" error instead of a misleading "unsupported value" error. + assertPlan(rel.getJoinType.nonEmpty, "NearestByJoin.join_type must be set") + assertPlan(rel.getMode.nonEmpty, "NearestByJoin.mode must be set") + assertPlan(rel.getDirection.nonEmpty, "NearestByJoin.direction must be set") + val left = Dataset.ofRows(session, transformRelation(rel.getLeft)) + val right = Dataset.ofRows(session, transformRelation(rel.getRight)) + val rankingExpression = Column(transformExpression(rel.getRankingExpression)) + left + .nearestByJoin( + right, + rankingExpression, + rel.getNumResults, + rel.getMode, + rel.getDirection, + rel.getJoinType) + .logicalPlan + } + private def transformSort(sort: proto.Sort): LogicalPlan = { assertPlan(sort.getOrderCount > 0, "'order' must be present and contain elements.") logical.Sort( diff --git a/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala b/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala index 91d51163b319e..d83a4df51cd52 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala @@ -764,6 +764,66 @@ class Dataset[T] private[sql]( lateralJoin(right, Some(joinExprs), LateralJoinType(joinType)) } + private[sql] def nearestByJoin( + right: sql.Dataset[_], + rankingExpression: Column, + numResults: Int, + joinType: JoinType, + approx: Boolean, + direction: NearestByDirection): DataFrame = { + if (numResults < 1 || numResults > NearestByJoin.MaxNumResults) { + throw new AnalysisException( + errorClass = "NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE", + messageParameters = Map( + "numResults" -> numResults.toString, + "min" -> "1", + "max" -> NearestByJoin.MaxNumResults.toString)) + } + withPlan { + NearestByJoin( + logicalPlan, + right.logicalPlan, + joinType, + approx, + numResults, + rankingExpression.expr, + direction) + } + } + + /** @inheritdoc */ + def nearestByJoin( + right: sql.Dataset[_], + rankingExpression: Column, + numResults: Int, + mode: String, + direction: String): DataFrame = { + nearestByJoin( + right, + rankingExpression, + numResults, + Inner, + NearestByJoinMode(mode), + NearestByDirection(direction)) + } + + /** @inheritdoc */ + def nearestByJoin( + right: sql.Dataset[_], + rankingExpression: Column, + numResults: Int, + mode: String, + direction: String, + joinType: String): DataFrame = { + nearestByJoin( + right, + rankingExpression, + numResults, + NearestByJoinType(joinType), + NearestByJoinMode(mode), + NearestByDirection(direction)) + } + // TODO(SPARK-22947): Fix the DataFrame API. private[sql] def joinAsOf( other: Dataset[_], diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameNearestByJoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameNearestByJoinSuite.scala new file mode 100644 index 0000000000000..b34880b71f5be --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameNearestByJoinSuite.scala @@ -0,0 +1,444 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql + +import org.apache.spark.sql.catalyst.plans.{NearestByDirection, NearestByJoinMode, NearestByJoinType} +import org.apache.spark.sql.execution.streaming.runtime.MemoryStream +import org.apache.spark.sql.functions._ +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.test.SharedSparkSession +import org.apache.spark.tags.SlowSQLTest + +@SlowSQLTest +class DataFrameNearestByJoinSuite extends QueryTest with SharedSparkSession { + + private def prepareForNearestByJoin(): (classic.DataFrame, classic.DataFrame) = { + val users = spark.createDataFrame( + Seq((1, 10.0), (2, 20.0), (3, 30.0))).toDF("user_id", "score") + val products = spark.createDataFrame( + Seq(("A", 11.0), ("B", 22.0), ("C", 5.0))).toDF("product", "pscore") + (users, products) + } + + test("similarity, inner, k=1") { + val (users, products) = prepareForNearestByJoin() + val result = users.nearestByJoin( + products, + -abs(users("score") - products("pscore")), + numResults = 1, + mode = "exact", + direction = "similarity") + + checkAnswer( + result.select("user_id", "product").orderBy("user_id"), + Seq(Row(1, "A"), Row(2, "B"), Row(3, "B")) + ) + } + + test("distance, inner, k=2") { + val (users, products) = prepareForNearestByJoin() + val result = users.nearestByJoin( + products, + abs(users("score") - products("pscore")), + numResults = 2, + mode = "exact", + direction = "distance") + + // For each user_id, closest 2 by |score - pscore|: + // user 1 (10): A (|10-11|=1), C (|10-5|=5) + // user 2 (20): B (|20-22|=2), A (|20-11|=9) + // user 3 (30): B (|30-22|=8), A (|30-11|=19) + checkAnswer( + result.select("user_id", "product").orderBy("user_id", "product"), + Seq( + Row(1, "A"), Row(1, "C"), + Row(2, "A"), Row(2, "B"), + Row(3, "A"), Row(3, "B")) + ) + } + + test("left outer when right side is empty") { + val (users, products) = prepareForNearestByJoin() + val emptyProducts = products.filter(lit(false)) + val result = users.nearestByJoin( + emptyProducts, + -abs(users("score") - emptyProducts("pscore")), + numResults = 1, + joinType = "leftouter", + mode = "approx", + direction = "similarity") + + checkAnswer( + result.select("user_id", "product").orderBy("user_id"), + Seq(Row(1, null), Row(2, null), Row(3, null)) + ) + } + + test("inner drops left rows with no matches") { + val (users, products) = prepareForNearestByJoin() + val emptyProducts = products.filter(lit(false)) + val result = users.nearestByJoin( + emptyProducts, + -abs(users("score") - emptyProducts("pscore")), + numResults = 1, + mode = "exact", + direction = "similarity") + + assert(result.count() === 0) + } + + test("self-join: each row finds nearest other rows in the same DataFrame") { + val (users, _) = prepareForNearestByJoin() + // We pass `users` as both sides; DeduplicateRelations rewrites the right side to + // generate fresh ExprIds, so the join resolves. Both `users("score")` references in + // the ranking expression bind to the original (left) attribute, so the rank is + // identically 0 for every candidate -- this test exercises self-join resolution, + // not nearest-row selection. + val result = users.nearestByJoin( + users, + -abs(users("score") - users("score")), + numResults = 2, + mode = "exact", + direction = "similarity") + + // 3 users x 2 nearest = 6 rows; output schema has user_id and score from both sides. + assert(result.count() === 6) + assert(result.columns.length === 4) + } + + test("inner: NULL ranking values for all candidates drops the left row") { + // Construct a left side where every comparison yields NULL: a NULL score on the left makes + // `abs(left.score - right.pscore)` evaluate to NULL for every right row, so MaxMinByK skips + // every candidate (its `ord == null` early-return path) and the heap stays empty. With INNER, + // the left row is dropped entirely. + val users = spark.createDataFrame( + Seq[(Int, java.lang.Double)]((1, null), (2, 20.0d))).toDF("user_id", "score") + val products = spark.createDataFrame( + Seq(("A", 11.0), ("B", 22.0))).toDF("product", "pscore") + + val result = users.nearestByJoin( + products, + abs(users("score") - products("pscore")), + numResults = 1, + mode = "exact", + direction = "distance") + + // Only user 2 should appear; user 1 (NULL score) drops because no candidate has a + // non-null ranking value. + checkAnswer( + result.select("user_id", "product"), + Seq(Row(2, "B")) + ) + } + + test("left outer: NULL ranking values for all candidates preserves left with NULLs") { + // Same shape as the previous test, but LEFT OUTER preserves user 1 with NULL right-side + // columns instead of dropping it. + val users = spark.createDataFrame( + Seq[(Int, java.lang.Double)]((1, null), (2, 20.0d))).toDF("user_id", "score") + val products = spark.createDataFrame( + Seq(("A", 11.0), ("B", 22.0))).toDF("product", "pscore") + + val result = users.nearestByJoin( + products, + abs(users("score") - products("pscore")), + numResults = 1, + joinType = "leftouter", + mode = "exact", + direction = "distance") + + checkAnswer( + result.select("user_id", "product").orderBy("user_id"), + Seq(Row(1, null), Row(2, "B")) + ) + } + + test("numResults larger than right side returns min(k, available) per left row") { + // Right side has 3 rows; ask for 5. Each left row should get exactly 3 matches, not 5 + // padded with NULLs. + val (users, products) = prepareForNearestByJoin() + val result = users.nearestByJoin( + products, + abs(users("score") - products("pscore")), + numResults = 5, + mode = "exact", + direction = "distance") + + // 3 users x min(5, 3) = 9 rows. + assert(result.count() === 9) + // No NULL padding: every left row pairs with every product exactly once. + val perUser = result.groupBy("user_id").count().collect().map(r => r.getInt(0) -> r.getLong(1)) + assert(perUser.toMap === Map(1 -> 3L, 2 -> 3L, 3 -> 3L)) + } + + test("duplicate left rows each get an independent top-K") { + // Two identical user rows must not be collapsed into a single group: each must independently + // produce its own top-K. This proves the per-row __qid tagging in the rewrite works. + val users = spark.createDataFrame( + Seq((1, 10.0), (1, 10.0))).toDF("user_id", "score") + val products = spark.createDataFrame( + Seq(("A", 11.0), ("B", 22.0), ("C", 5.0))).toDF("product", "pscore") + + val result = users.nearestByJoin( + products, + abs(users("score") - products("pscore")), + numResults = 1, + mode = "exact", + direction = "distance") + + // Two identical left rows -> two output rows, both pairing with product A (closest to 10.0). + checkAnswer( + result.select("user_id", "product"), + Seq(Row(1, "A"), Row(1, "A")) + ) + } + + test("conflicting column names between sides resolve via DataFrame qualifiers") { + // Both sides have a column named `score`; the ranking expression disambiguates via + // DataFrame-qualified accessors. + val left = spark.createDataFrame(Seq((1, 10.0), (2, 20.0))).toDF("id", "score") + val right = spark.createDataFrame( + Seq(("A", 11.0), ("B", 22.0), ("C", 5.0))).toDF("name", "score") + + val result = left.nearestByJoin( + right, + -abs(left("score") - right("score")), + numResults = 1, + mode = "exact", + direction = "similarity") + + checkAnswer( + result.select("id", "name").orderBy("id"), + Seq(Row(1, "A"), Row(2, "B")) + ) + // Output schema should carry both `score` columns through (4 columns total). + assert(result.columns.length === 4) + } + + test("streaming inputs are rejected at analysis time") { + // Build a streaming left side and a static right side; NearestByJoin must be rejected + // at analysis before the optimizer rewrite (an unconditioned cross-product fed into a + // global Aggregate keyed by a per-row identifier) ever runs. + import testImplicits._ + implicit val ctx = spark.sqlContext + val streamingUsers = MemoryStream[(Int, Double)].toDF().toDF("user_id", "score") + val products = spark.createDataFrame( + Seq(("A", 11.0), ("B", 22.0), ("C", 5.0))).toDF("product", "pscore") + + checkError( + exception = intercept[AnalysisException] { + streamingUsers.nearestByJoin( + products, + -abs(streamingUsers("score") - products("pscore")), + numResults = 1, + mode = "exact", + direction = "similarity").queryExecution.analyzed + }, + condition = "NEAREST_BY_JOIN.STREAMING_NOT_SUPPORTED", + parameters = Map.empty) + } + + test("rejected when spark.sql.crossJoin.enabled is false") { + // The rewrite produces an unconditioned cross-product internally, so when the user has + // opted out of cross-products via `spark.sql.crossJoin.enabled = false`, NEAREST BY + // queries are rejected at analysis time with `NEAREST_BY_JOIN.CROSS_JOIN_NOT_ENABLED` -- + // a NEAREST BY-specific error class added so the user does not see internal rewrite + // attributes in the error message. + withSQLConf(SQLConf.CROSS_JOINS_ENABLED.key -> "false") { + val (users, products) = prepareForNearestByJoin() + checkError( + exception = intercept[AnalysisException] { + users.nearestByJoin( + products, + -abs(users("score") - products("pscore")), + numResults = 1, + mode = "exact", + direction = "similarity").queryExecution.analyzed + }, + condition = "NEAREST_BY_JOIN.CROSS_JOIN_NOT_ENABLED", + parameters = Map.empty) + } + } + + test("exact + left outer: empty right side preserves all left rows with NULLs") { + // Exercises the EXACT + LEFT OUTER combination, which no other test covers together. + val (users, products) = prepareForNearestByJoin() + val emptyProducts = products.filter(lit(false)) + val result = users.nearestByJoin( + emptyProducts, + -abs(users("score") - emptyProducts("pscore")), + numResults = 1, + joinType = "leftouter", + mode = "exact", + direction = "similarity") + + checkAnswer( + result.select("user_id", "product").orderBy("user_id"), + Seq(Row(1, null), Row(2, null), Row(3, null)) + ) + } + + test("SQL: APPROX NEAREST SIMILARITY") { + val (users, products) = prepareForNearestByJoin() + users.createOrReplaceTempView("t_users") + products.createOrReplaceTempView("t_products") + try { + val result = spark.sql( + """ + |SELECT u.user_id, p.product + |FROM t_users u JOIN t_products p + | APPROX NEAREST 1 BY SIMILARITY -abs(u.score - p.pscore) + |""".stripMargin) + checkAnswer( + result.orderBy("user_id"), + Seq(Row(1, "A"), Row(2, "B"), Row(3, "B")) + ) + } finally { + spark.catalog.dropTempView("t_users") + spark.catalog.dropTempView("t_products") + } + } + + test("SQL: EXACT NEAREST DISTANCE") { + val (users, products) = prepareForNearestByJoin() + users.createOrReplaceTempView("t_users") + products.createOrReplaceTempView("t_products") + try { + val result = spark.sql( + """ + |SELECT u.user_id, p.product + |FROM t_users u JOIN t_products p + | EXACT NEAREST 1 BY DISTANCE abs(u.score - p.pscore) + |""".stripMargin) + checkAnswer( + result.orderBy("user_id"), + Seq(Row(1, "A"), Row(2, "B"), Row(3, "B")) + ) + } finally { + spark.catalog.dropTempView("t_users") + spark.catalog.dropTempView("t_products") + } + } + + test("invalid numResults is rejected") { + val (users, products) = prepareForNearestByJoin() + Seq(0, 100001).foreach { k => + checkError( + exception = intercept[AnalysisException] { + users.nearestByJoin( + products, + -abs(users("score") - products("pscore")), + numResults = k, + mode = "exact", + direction = "similarity") + }, + condition = "NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE", + parameters = Map( + "numResults" -> k.toString, + "min" -> "1", + "max" -> "100000")) + } + } + + test("invalid joinType is rejected") { + val (users, products) = prepareForNearestByJoin() + checkError( + exception = intercept[AnalysisException] { + users.nearestByJoin( + products, + -abs(users("score") - products("pscore")), + numResults = 1, + joinType = "rightouter", + mode = "approx", + direction = "similarity") + }, + condition = "NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE", + parameters = Map( + "joinType" -> "rightouter", + "supported" -> NearestByJoinType.supportedDisplay)) + } + + test("invalid mode is rejected") { + val (users, products) = prepareForNearestByJoin() + checkError( + exception = intercept[AnalysisException] { + users.nearestByJoin( + products, + -abs(users("score") - products("pscore")), + numResults = 1, + joinType = "inner", + mode = "bogus", + direction = "similarity") + }, + condition = "NEAREST_BY_JOIN.UNSUPPORTED_MODE", + parameters = Map( + "mode" -> "bogus", + "supported" -> NearestByJoinMode.supported.mkString("'", "', '", "'"))) + } + + test("invalid direction is rejected") { + val (users, products) = prepareForNearestByJoin() + checkError( + exception = intercept[AnalysisException] { + users.nearestByJoin( + products, + -abs(users("score") - products("pscore")), + numResults = 1, + mode = "exact", + direction = "bogus") + }, + condition = "NEAREST_BY_JOIN.UNSUPPORTED_DIRECTION", + parameters = Map( + "direction" -> "bogus", + "supported" -> NearestByDirection.supported.mkString("'", "', '", "'"))) + } + + test("non-orderable ranking expression is rejected") { + val (users, products) = prepareForNearestByJoin() + checkError( + exception = intercept[AnalysisException] { + users.nearestByJoin( + products, + map(users("score"), products("pscore")), + numResults = 1, + mode = "exact", + direction = "similarity") + }, + condition = "NEAREST_BY_JOIN.NON_ORDERABLE_RANKING_EXPRESSION", + parameters = Map( + "expression" -> "\"map(score, pscore)\"", + "type" -> "\"MAP\"")) + } + + test("EXACT mode rejects nondeterministic ranking expression") { + val (users, products) = prepareForNearestByJoin() + checkError( + exception = intercept[AnalysisException] { + users.nearestByJoin( + products, + rand() + products("pscore"), + numResults = 1, + joinType = "inner", + mode = "exact", + direction = "similarity") + }, + condition = "NEAREST_BY_JOIN.EXACT_WITH_NONDETERMINISTIC_EXPRESSION", + matchPVals = true, + parameters = Map("expression" -> ".*rand.*pscore.*")) + } +} From 75580b2d3ad908dacf55cfaa9cb644d980ab1337 Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Thu, 14 May 2026 16:58:58 +0800 Subject: [PATCH 120/286] [SPARK-56395][SQL][FOLLOWUP] Fix missing comma in MimaExcludes on branch-4.2 ### What changes were proposed in this pull request? Add a missing trailing comma to the `DataStreamReader.name` MiMa exclude entry in `project/MimaExcludes.scala` on branch-4.2. The cherry-pick of SPARK-56395 (commit 2c356ac6c2d) added a new `Dataset.nearestByJoin` exclude entry to `v42excludes` but did not append the required comma to the preceding `DataStreamReader.name` entry. ### Why are the changes needed? Without the comma, SBT project loading fails on branch-4.2: ``` [error] /home/runner/work/spark/spark/project/MimaExcludes.scala:62:19: ')' expected but '.' found. [error] ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.Dataset.nearestByJoin") [error] ^ [error] /home/runner/work/spark/spark/project/MimaExcludes.scala:63:3: ';' expected but ')' found. [error] ) [error] ^ [error] two errors found [error] (Compile / compileIncremental) Compilation failed ``` This breaks all CI builds on branch-4.2. Reported by LuciferYang in https://github.com/apache/spark/pull/55682#discussion_r3239241097. This change is branch-4.2 only; master is unaffected because the `nearestByJoin` entry has not been merged to master yet. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing CI. The fix restores compilability of the SBT project definition. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes #55873 from cloud-fan/cloud-fan/SPARK-56395-branch-4.2-fix. Authored-by: Wenchen Fan Signed-off-by: Wenchen Fan --- project/MimaExcludes.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index ec9146459d8d4..6bdd04517f06f 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -57,7 +57,7 @@ object MimaExcludes { // [SPARK-56330][CORE] Add TaskInterruptListener to TaskContext for interrupt notifications ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.addTaskInterruptListener"), // [SPARK-56700][SS] Make DataStreamReader.name public - ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.streaming.DataStreamReader.name") + ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.streaming.DataStreamReader.name"), // [SPARK-56395][SQL] Add NEAREST BY top-K ranking join ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.Dataset.nearestByJoin") ) From 37d57984782c37f125ff34d51ee58359f0b87361 Mon Sep 17 00:00:00 2001 From: Chao Sun Date: Thu, 14 May 2026 11:18:47 +0200 Subject: [PATCH 121/286] [SPARK-56840][SQL] Avoid unresolved NullIf type lookup ### Why are the changes needed? `NULLIF` builds its replacement expression before analysis has resolved all child expressions. For nested field references, the existing implementation can read the left operand's data type too early while constructing the null branch, which can fail analysis even though the SQL shape is valid. SPARK-56840 tracks this analyzer failure. ### What changes were proposed in this PR? - Build the `NULLIF` null branch with a lazy typed-null placeholder so construction does not eagerly read the unresolved left operand type, while `NullIf.replacement.dataType` remains valid once the operand type is available. - Make that placeholder `RuntimeReplaceable`, so `ReplaceExpressions` restores an ordinary typed `Literal(null, ...)` before later optimizer rules run and existing null-literal simplifications continue to apply. - Add focused regressions for: - nested struct-field `nullif(c.provider, lower(...))` analysis in both `ALWAYS_INLINE_COMMON_EXPR` modes; - `NullIf` replacement type reporting before type coercion; - optimizer replacement back to a normal null literal; - explain output avoiding exposure of the internal helper name. ### Does this PR introduce _any_ user-facing change? Yes. Valid `NULLIF` expressions over unresolved nested field references that could fail during analysis now resolve and execute successfully. ### How was this patch tested? - `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.NullExpressionsSuite -- -z "NullIf replacement preserves its data type before type coercion"'` - `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizerSuite -- -z "NullIf typed null branch is replaced with a null literal"'` - `build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z "nullif function"'` - `build/sbt 'sql/testOnly org.apache.spark.sql.ExplainSuite -- -z "explain for these functions; use range to avoid constant folding"'` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5.5) Closes #55838 from sunchao/dev/chao/codex/oss-nullif-unresolved. Authored-by: Chao Sun Signed-off-by: Peter Toth (cherry picked from commit 5949ab30b41860574ab57b94a8848464b5e127a7) Signed-off-by: Peter Toth --- .../expressions/nullExpressions.scala | 19 +++++++++++++-- .../expressions/NullExpressionsSuite.scala | 10 ++++++++ .../catalyst/optimizer/OptimizerSuite.scala | 24 +++++++++++++++++-- .../spark/sql/DataFrameFunctionsSuite.scala | 7 ++++++ .../org/apache/spark/sql/ExplainSuite.scala | 1 + 5 files changed, 57 insertions(+), 4 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala index 1aa1d0b25e44c..e7be588c4b465 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala @@ -145,6 +145,21 @@ case class Coalesce(children: Seq[Expression]) copy(children = newChildren) } +private case class TypedNullLiteral(child: Expression) + extends UnaryExpression with RuntimeReplaceable { + override def nullable: Boolean = true + + override def dataType: DataType = child.dataType + + override def toString: String = "null" + + override def sql: String = "NULL" + + override lazy val replacement: Expression = Literal.create(null, child.dataType) + + override protected def withNewChildInternal(newChild: Expression): TypedNullLiteral = + copy(child = newChild) +} @ExpressionDescription( usage = "_FUNC_(expr1, expr2) - Returns null if `expr1` equals to `expr2`, or `expr1` otherwise.", @@ -162,10 +177,10 @@ case class NullIf(left: Expression, right: Expression, replacement: Expression) this(left, right, if (!SQLConf.get.getConf(SQLConf.ALWAYS_INLINE_COMMON_EXPR)) { With(left) { case Seq(ref) => - If(EqualTo(ref, right), Literal.create(null, left.dataType), ref) + If(EqualTo(ref, right), TypedNullLiteral(ref), ref) } } else { - If(EqualTo(left, right), Literal.create(null, left.dataType), left) + If(EqualTo(left, right), TypedNullLiteral(left), left) } ) } diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala index c74a9e35833d1..bb4aed9b40021 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala @@ -143,6 +143,16 @@ class NullExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper { assert(analyze(new Nvl(floatLit, doubleLit)).dataType == DoubleType) } + test("NullIf replacement preserves its data type before type coercion") { + Seq(true, false).foreach { alwaysInlineCommonExpr => + withSQLConf(SQLConf.ALWAYS_INLINE_COMMON_EXPR.key -> alwaysInlineCommonExpr.toString) { + val nullIf = new NullIf(Literal(1), Literal(1)) + assert(nullIf.dataType == IntegerType) + assert(nullIf.replacement.dataType == IntegerType) + } + } + } + test("AtLeastNNonNulls") { val mix = Seq(Literal("x"), Literal.create(null, StringType), diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerSuite.scala index 70a2ae94109fc..057e4ceaf0a01 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerSuite.scala @@ -21,13 +21,13 @@ import org.apache.spark.SparkException import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.dsl.expressions._ import org.apache.spark.sql.catalyst.dsl.plans._ -import org.apache.spark.sql.catalyst.expressions.{Add, Alias, ArrayCompact, AttributeReference, CreateArray, CreateStruct, IntegerLiteral, Literal, MapFromEntries, Multiply, NamedExpression, Remainder} +import org.apache.spark.sql.catalyst.expressions.{Add, Alias, ArrayCompact, AttributeReference, CreateArray, CreateStruct, IntegerLiteral, Literal, MapFromEntries, Multiply, NamedExpression, NullIf, Remainder, RuntimeReplaceable} import org.apache.spark.sql.catalyst.expressions.aggregate.Sum import org.apache.spark.sql.catalyst.plans.PlanTest import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LocalRelation, LogicalPlan, OneRowRelation, Project} import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.internal.SQLConf -import org.apache.spark.sql.types.{ArrayType, IntegerType, MapType, StructField, StructType} +import org.apache.spark.sql.types.{ArrayType, BooleanType, IntegerType, MapType, StructField, StructType} /** * A dummy optimizer rule for testing that decrements integer literals until 0. @@ -334,4 +334,24 @@ class OptimizerSuite extends PlanTest { assert(optimized2.schema === StructType(StructField("map", MapType(IntegerType, IntegerType, false), false) :: Nil)) } + + test("NullIf typed null branch is replaced with a null literal") { + val optimizer = new SimpleTestOptimizer() { + override def defaultBatches: Seq[Batch] = + Batch("test", fixedPoint, + ReplaceExpressions) :: Nil + } + + withSQLConf(SQLConf.ALWAYS_INLINE_COMMON_EXPR.key -> "true") { + val nullIf = new NullIf(Literal(true), Literal(true)) + val plan = Project(Alias(nullIf, "out")() :: Nil, OneRowRelation()).analyze + val optimized = optimizer.execute(plan) + + assert(optimized.expressions.exists(_.exists { + case Literal(null, BooleanType) => true + case _ => false + })) + assert(optimized.expressions.forall(!_.exists(_.isInstanceOf[RuntimeReplaceable]))) + } + } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala index a0d9d2e9f40d3..7faccbde997dd 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala @@ -350,6 +350,13 @@ class DataFrameFunctionsSuite extends SharedSparkSession { "expression" -> "\"id\"", "expressionAnyValue" -> "\"any_value(id)\"") ) + + val nestedDf = Seq("error_multiple_providers", "openai") + .toDF("provider") + .select(struct(col("provider")).as("c")) + checkAnswer( + nestedDf.select(nullif(col("c.provider"), lower(lit("ERROR_MULTIPLE_PROVIDERS")))), + Seq(Row(null), Row("openai"))) } } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala index 04be1e8fcfba3..af52204dbb7ff 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala @@ -251,6 +251,7 @@ class ExplainSuite extends ExplainSuiteHelper with DisableAdaptiveExecutionSuite checkKeywordsExistsInExplain(df, "Project [id#xL AS ifnull(id, 1)#xL, if ((id#xL = 1)) null " + "else id#xL AS nullif(id, 1)#xL, id#xL AS nvl(id, 1)#xL, 1 AS nvl2(id, 1, 2)#x]") + checkKeywordsNotExistsInExplain(df, ExtendedMode, "typednullliteral") } test("SPARK-26659: explain of DataWritingCommandExec should not contain duplicate cmd.nodeName") { From edebd6315e99f03f862ae0acf7fc06ef32c17d42 Mon Sep 17 00:00:00 2001 From: YangJie Date: Thu, 14 May 2026 18:58:26 +0200 Subject: [PATCH 122/286] [SPARK-56817][BUILD][4.2] Upgrade Netty to 4.2.13.Final ### What changes were proposed in this pull request? This PR upgrades `Netty` to 4.2.13.Final. ### Why are the changes needed? This version includes the 11 CVE fixes: - [CVE-2026-42586](https://github.com/netty/netty/security/advisories/GHSA-rgrr-p7gp-5xj7) (netty-codec-redis) - [CVE-2026-42578](https://github.com/netty/netty/security/advisories/GHSA-45q3-82m4-75jr) (netty-handler-proxy) - [CVE-2026-42577](https://github.com/netty/netty/security/advisories/GHSA-rwm7-x88c-3g2p) (netty-transport-native-epoll) - [CVE-2026-42587](https://github.com/netty/netty/security/advisories/GHSA-f6hv-jmp6-3vwv) (netty-codec-http, netty-codec-http2) - [CVE-2026-41417](https://github.com/netty/netty/security/advisories/GHSA-v8h7-rr48-vmmv) (netty-codec-http) - [CVE-2026-42581](https://github.com/netty/netty/security/advisories/GHSA-xxqh-mfjm-7mv9) (netty-codec-http) - [CVE-2026-42580](https://github.com/netty/netty/security/advisories/GHSA-m4cv-j2px-7723) (netty-codec-http) - [CVE-2026-42585](https://github.com/netty/netty/security/advisories/GHSA-38f8-5428-x5cv) (netty-codec-http) - [CVE-2026-42579](https://github.com/netty/netty/security/advisories/GHSA-cm33-6792-r9fm) (netty-codec-dns) - [CVE-2026-42582](https://github.com/netty/netty/security/advisories/GHSA-2c5c-chwr-9hqw) (netty-codec-http3) - [CVE-2026-42583](https://github.com/netty/netty/security/advisories/GHSA-mj4r-2hfc-f8p6) (netty-codec, netty-codec-compression) - [CVE-2026-42584](https://github.com/netty/netty/security/advisories/GHSA-57rv-r2g8-2cj3) (netty-codec-http) - [CVE-2026-44248](https://github.com/netty/netty/security/advisories/GHSA-jfg9-48mv-9qgx) (netty-codec-mqtt) At least the following issues may have affected Apache Spark: - https://github.com/apache/spark/security/dependabot/187 The full release notes as follows: - https://netty.io/news/2026/05/04/4-2-13-Final.html ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass Github Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #55859 from LuciferYang/SPARK-56817-4.2. Authored-by: YangJie Signed-off-by: Peter Toth --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 46 +++++++++++++-------------- pom.xml | 2 +- 2 files changed, 24 insertions(+), 24 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index b45e4ea858d47..fbf8f06a34046 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -196,35 +196,35 @@ metrics-jmx/4.2.37//metrics-jmx-4.2.37.jar metrics-json/4.2.37//metrics-json-4.2.37.jar metrics-jvm/4.2.37//metrics-jvm-4.2.37.jar minlog/1.3.0//minlog-1.3.0.jar -netty-all/4.2.12.Final//netty-all-4.2.12.Final.jar -netty-buffer/4.2.12.Final//netty-buffer-4.2.12.Final.jar -netty-codec-base/4.2.12.Final//netty-codec-base-4.2.12.Final.jar -netty-codec-compression/4.2.12.Final//netty-codec-compression-4.2.12.Final.jar -netty-codec-dns/4.2.12.Final//netty-codec-dns-4.2.12.Final.jar -netty-codec-http/4.2.12.Final//netty-codec-http-4.2.12.Final.jar -netty-codec-http2/4.2.12.Final//netty-codec-http2-4.2.12.Final.jar -netty-codec-socks/4.2.12.Final//netty-codec-socks-4.2.12.Final.jar -netty-codec/4.2.12.Final//netty-codec-4.2.12.Final.jar -netty-common/4.2.12.Final//netty-common-4.2.12.Final.jar -netty-handler-proxy/4.2.12.Final//netty-handler-proxy-4.2.12.Final.jar -netty-handler/4.2.12.Final//netty-handler-4.2.12.Final.jar -netty-resolver-dns/4.2.12.Final//netty-resolver-dns-4.2.12.Final.jar -netty-resolver/4.2.12.Final//netty-resolver-4.2.12.Final.jar +netty-all/4.2.13.Final//netty-all-4.2.13.Final.jar +netty-buffer/4.2.13.Final//netty-buffer-4.2.13.Final.jar +netty-codec-base/4.2.13.Final//netty-codec-base-4.2.13.Final.jar +netty-codec-compression/4.2.13.Final//netty-codec-compression-4.2.13.Final.jar +netty-codec-dns/4.2.13.Final//netty-codec-dns-4.2.13.Final.jar +netty-codec-http/4.2.13.Final//netty-codec-http-4.2.13.Final.jar +netty-codec-http2/4.2.13.Final//netty-codec-http2-4.2.13.Final.jar +netty-codec-socks/4.2.13.Final//netty-codec-socks-4.2.13.Final.jar +netty-codec/4.2.13.Final//netty-codec-4.2.13.Final.jar +netty-common/4.2.13.Final//netty-common-4.2.13.Final.jar +netty-handler-proxy/4.2.13.Final//netty-handler-proxy-4.2.13.Final.jar +netty-handler/4.2.13.Final//netty-handler-4.2.13.Final.jar +netty-resolver-dns/4.2.13.Final//netty-resolver-dns-4.2.13.Final.jar +netty-resolver/4.2.13.Final//netty-resolver-4.2.13.Final.jar netty-tcnative-boringssl-static/2.0.76.Final/linux-aarch_64/netty-tcnative-boringssl-static-2.0.76.Final-linux-aarch_64.jar netty-tcnative-boringssl-static/2.0.76.Final/linux-x86_64/netty-tcnative-boringssl-static-2.0.76.Final-linux-x86_64.jar netty-tcnative-boringssl-static/2.0.76.Final/osx-aarch_64/netty-tcnative-boringssl-static-2.0.76.Final-osx-aarch_64.jar netty-tcnative-boringssl-static/2.0.76.Final/osx-x86_64/netty-tcnative-boringssl-static-2.0.76.Final-osx-x86_64.jar netty-tcnative-boringssl-static/2.0.76.Final/windows-x86_64/netty-tcnative-boringssl-static-2.0.76.Final-windows-x86_64.jar netty-tcnative-classes/2.0.76.Final//netty-tcnative-classes-2.0.76.Final.jar -netty-transport-classes-epoll/4.2.12.Final//netty-transport-classes-epoll-4.2.12.Final.jar -netty-transport-classes-kqueue/4.2.12.Final//netty-transport-classes-kqueue-4.2.12.Final.jar -netty-transport-native-epoll/4.2.12.Final/linux-aarch_64/netty-transport-native-epoll-4.2.12.Final-linux-aarch_64.jar -netty-transport-native-epoll/4.2.12.Final/linux-riscv64/netty-transport-native-epoll-4.2.12.Final-linux-riscv64.jar -netty-transport-native-epoll/4.2.12.Final/linux-x86_64/netty-transport-native-epoll-4.2.12.Final-linux-x86_64.jar -netty-transport-native-kqueue/4.2.12.Final/osx-aarch_64/netty-transport-native-kqueue-4.2.12.Final-osx-aarch_64.jar -netty-transport-native-kqueue/4.2.12.Final/osx-x86_64/netty-transport-native-kqueue-4.2.12.Final-osx-x86_64.jar -netty-transport-native-unix-common/4.2.12.Final//netty-transport-native-unix-common-4.2.12.Final.jar -netty-transport/4.2.12.Final//netty-transport-4.2.12.Final.jar +netty-transport-classes-epoll/4.2.13.Final//netty-transport-classes-epoll-4.2.13.Final.jar +netty-transport-classes-kqueue/4.2.13.Final//netty-transport-classes-kqueue-4.2.13.Final.jar +netty-transport-native-epoll/4.2.13.Final/linux-aarch_64/netty-transport-native-epoll-4.2.13.Final-linux-aarch_64.jar +netty-transport-native-epoll/4.2.13.Final/linux-riscv64/netty-transport-native-epoll-4.2.13.Final-linux-riscv64.jar +netty-transport-native-epoll/4.2.13.Final/linux-x86_64/netty-transport-native-epoll-4.2.13.Final-linux-x86_64.jar +netty-transport-native-kqueue/4.2.13.Final/osx-aarch_64/netty-transport-native-kqueue-4.2.13.Final-osx-aarch_64.jar +netty-transport-native-kqueue/4.2.13.Final/osx-x86_64/netty-transport-native-kqueue-4.2.13.Final-osx-x86_64.jar +netty-transport-native-unix-common/4.2.13.Final//netty-transport-native-unix-common-4.2.13.Final.jar +netty-transport/4.2.13.Final//netty-transport-4.2.13.Final.jar objenesis/3.5//objenesis-3.5.jar okhttp/3.12.12//okhttp-3.12.12.jar okio/1.17.6//okio-1.17.6.jar diff --git a/pom.xml b/pom.xml index 57faf6d781997..efc29699b93e5 100644 --- a/pom.xml +++ b/pom.xml @@ -220,7 +220,7 @@ SPARK-53327 workaround should be reverted. --> 6.2.0 - 4.2.12.Final + 4.2.13.Final 2.0.76.Final 78.3 6.0.3 From 17f9b9d1561da311a205f1befe9edd1f720769c2 Mon Sep 17 00:00:00 2001 From: Mark Jarvin Date: Fri, 15 May 2026 01:35:59 +0800 Subject: [PATCH 123/286] [SPARK-56756][SQL] Add error class for recursiveFileLookup conflict with partitioned data source ### What changes were proposed in this pull request? `PartitioningAwareFileIndex.listFiles` rejects the combination of `recursiveFileLookup=true` and a non-empty `partitionSpec().partitionColumns` by throwing a raw `java.lang.IllegalArgumentException` with the message "Datasource with partition do not allow recursive file loading." This PR replaces that with a tagged `AnalysisException` using a new error class: - New error class `RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE` (`sqlState 0A000`) in `error-conditions.json`. - New helper `QueryCompilationErrors.recursiveFileLookupNotSupportedForPartitionedDataSourceError()`. - Throw site in `PartitioningAwareFileIndex.scala` updated to use the helper. ### Why are the changes needed? The raw `IllegalArgumentException` is unclassified and does not surface as a user-facing error with a clear message. Replacing it with an `AnalysisException` using a proper error class ensures it is correctly classified as a user error with an actionable message. ### Does this PR introduce _any_ user-facing change? Yes. Users who hit this error will now see a clearer message: > Recursive file loading is not supported when the data source has explicit partition columns. Either remove the option "recursiveFileLookup", or read the data without supplying partition columns (for example, do not read a partitioned table or set partition-column options such as "cloudFiles.partitionColumns"). Previously the error was a raw `IllegalArgumentException` with the message "Datasource with partition do not allow recursive file loading." ### How was this patch tested? Added `"recursiveFileLookup with a partitioned catalog table is rejected"` in `FileBasedDataSourceSuite`, which creates a partitioned Parquet catalog table, then asserts that reading it with `recursiveFileLookup=true` throws an `AnalysisException` with condition `RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (claude-sonnet-4-6) Closes #55721 from markj-db/recursive-file-lookup-error-class. Authored-by: Mark Jarvin Signed-off-by: Wenchen Fan (cherry picked from commit 051ea2b17989a2f22c287d4de22af0537e033cff) Signed-off-by: Wenchen Fan --- .../resources/error/error-conditions.json | 6 +++++ .../sql/errors/QueryCompilationErrors.scala | 6 +++++ .../execution/datasources/DataSource.scala | 5 ++++ .../PartitioningAwareFileIndex.scala | 4 +-- .../spark/sql/FileBasedDataSourceSuite.scala | 23 +++++++++++++++++ .../datasources/FileIndexSuite.scala | 25 ++++++++++++++++++- 6 files changed, 66 insertions(+), 3 deletions(-) diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json index 889ecf9f7b08a..31421987e5030 100644 --- a/common/utils/src/main/resources/error/error-conditions.json +++ b/common/utils/src/main/resources/error/error-conditions.json @@ -6033,6 +6033,12 @@ ], "sqlState" : "42836" }, + "RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE" : { + "message" : [ + "Recursive file loading is not supported when the data source has explicit partition columns. Either remove the option \"recursiveFileLookup\", or read the data without supplying partition columns (for example, do not read a partitioned table)." + ], + "sqlState" : "0A000" + }, "RECURSIVE_PROTOBUF_SCHEMA" : { "message" : [ "Found recursive reference in Protobuf schema, which can not be processed by Spark by default: . try setting the option `recursive.fields.max.depth` 1 to 10. Going beyond 10 levels of recursion is not allowed." diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala index ceb384b2f533d..5cfdbc66e3f42 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala @@ -3440,6 +3440,12 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat "newPath" -> newPath.map(toSQLId).mkString(" -> "))) } + def recursiveFileLookupNotSupportedForPartitionedDataSourceError(): Throwable = { + new AnalysisException( + errorClass = "RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE", + messageParameters = Map.empty) + } + def notAllowedToCreatePermanentViewWithoutAssigningAliasForExpressionError( viewNameParts: Seq[String], attr: Attribute): Throwable = { diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala index 9b51d3763abba..4a95f681fb6e5 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala @@ -420,6 +420,11 @@ case class DataSource( catalogTable.isDefined && catalogTable.get.tracksPartitionsInCatalog && catalogTable.get.partitionColumnNames.nonEmpty val (fileCatalog, dataSchema, partitionSchema) = if (useCatalogFileIndex) { + if (caseInsensitiveOptions.getOrElse( + FileIndexOptions.RECURSIVE_FILE_LOOKUP, "false").toBoolean) { + throw QueryCompilationErrors + .recursiveFileLookupNotSupportedForPartitionedDataSourceError() + } val defaultTableSize = conf.defaultSizeInBytes val index = new CatalogFileIndex( sparkSession, diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala index 1bf0d2f0301f2..8cea2c95e6940 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala @@ -31,6 +31,7 @@ import org.apache.spark.sql.SparkSession import org.apache.spark.sql.catalyst.{expressions, InternalRow} import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap +import org.apache.spark.sql.errors.QueryCompilationErrors import org.apache.spark.sql.types.StructType import org.apache.spark.util.ArrayImplicits._ @@ -89,8 +90,7 @@ abstract class PartitioningAwareFileIndex( PartitionDirectory(InternalRow.empty, allFiles().toArray.filter(isNonEmptyFile))) :: Nil } else { if (recursiveFileLookup) { - throw new IllegalArgumentException( - "Datasource with partition do not allow recursive file loading.") + throw QueryCompilationErrors.recursiveFileLookupNotSupportedForPartitionedDataSourceError() } prunePartitions(partitionFilters, partitionSpec()).map { case PartitionPath(values, path) => diff --git a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala index 8aa6f5a5d0e6e..1fc45e9703f9e 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala @@ -878,6 +878,29 @@ class FileBasedDataSourceSuite extends SharedSparkSession assert(fileList.toSet === expectedFileList.toSet) } + test("recursiveFileLookup with a partitioned catalog table is rejected") { + withTable("part_tbl") { + sql( + """ + |CREATE TABLE part_tbl (id INT, value STRING) + |USING parquet + |PARTITIONED BY (year INT) + |""".stripMargin) + sql("INSERT INTO part_tbl PARTITION (year = 2024) VALUES (1, 'a')") + sql("INSERT INTO part_tbl PARTITION (year = 2025) VALUES (2, 'b')") + checkError( + exception = intercept[AnalysisException] { + spark.read + .option("recursiveFileLookup", "true") + .table("part_tbl") + .collect() + }, + condition = "RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE", + parameters = Map.empty[String, String] + ) + } + } + test("Return correct results when data columns overlap with partition columns") { Seq("parquet", "orc", "json").foreach { format => withTempPath { path => diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala index 1150f6163b978..f4de8a52810e4 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala @@ -31,7 +31,8 @@ import org.mockito.Mockito.{mock, when} import org.apache.spark.{SparkException, SparkRuntimeException} import org.apache.spark.metrics.source.HiveCatalogMetrics -import org.apache.spark.sql.{Row, SparkSession} +import org.apache.spark.sql.{AnalysisException, Row, SparkSession} +import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.util._ import org.apache.spark.sql.functions.col import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf} @@ -657,6 +658,28 @@ class FileIndexSuite extends SharedSparkSession { assert(FileIndexOptions.isValidOption("pathglobfilter")) } + test("recursiveFileLookup with a user-specified partition spec is rejected") { + withTempDir { dir => + val partitionSchema = StructType(Seq(StructField("year", IntegerType, nullable = true))) + val partitionSpec = PartitionSpec( + partitionSchema, + Seq(PartitionPath(InternalRow(2024), new Path(dir.getCanonicalPath)))) + val fileIndex = new InMemoryFileIndex( + spark, + rootPathsSpecified = Seq(new Path(dir.getCanonicalPath)), + parameters = Map("recursiveFileLookup" -> "true"), + userSpecifiedSchema = None, + userSpecifiedPartitionSpec = Some(partitionSpec)) + checkError( + exception = intercept[AnalysisException] { + fileIndex.listFiles(Nil, Nil) + }, + condition = "RECURSIVE_FILE_LOOKUP_NOT_SUPPORTED_FOR_PARTITIONED_DATA_SOURCE", + parameters = Map.empty[String, String] + ) + } + } + test("SPARK-52339: Correctly compare root paths") { withTempDir { dir => val file1 = new File(dir, "text1.txt") From e5e86781d7e4650485d71182a39254c87e8cfdb7 Mon Sep 17 00:00:00 2001 From: Uros Bojanic Date: Fri, 15 May 2026 01:45:55 +0800 Subject: [PATCH 124/286] [SPARK-56152][SQL] Enable implicit cast from STRING to TIME type ### What changes were proposed in this pull request? Enable implicit casting from StringType to TimeType. ### Why are the changes needed? Ensure proper interoperability of TIME with other data types. ### Does this PR introduce _any_ user-facing change? Yes, strings can now be implicitly casted to TimeType. ### How was this patch tested? Added appropriate unit tests and end-to-end SQL tests. ### Was this patch authored or co-authored using generative AI tooling? Yes, Claude Opus 4.7. Closes #54950 from uros-db/cast-string-to-time. Authored-by: Uros Bojanic Signed-off-by: Wenchen Fan (cherry picked from commit b74b9f737239292e75a0fad4a085b032a8ae6906) Signed-off-by: Wenchen Fan --- .../catalyst/analysis/AnsiTypeCoercion.scala | 4 +-- .../sql/catalyst/analysis/TypeCoercion.scala | 1 + .../analysis/AnsiTypeCoercionSuite.scala | 1 + .../catalyst/analysis/TypeCoercionSuite.scala | 10 ++++++ .../native/implicitTypeCasts.sql.out | 33 +++++++++++++++++++ .../typeCoercion/native/implicitTypeCasts.sql | 5 +++ .../native/implicitTypeCasts.sql.out | 24 ++++++++++++++ 7 files changed, 76 insertions(+), 2 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala index e23e7561f0e36..23c416dd4b383 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala @@ -192,8 +192,8 @@ object AnsiTypeCoercion extends TypeCoercionBase { // Ideally the implicit cast rule should be the same as `Cast.canANSIStoreAssign` so that it's // consistent with table insertion. To avoid breaking too many existing Spark SQL queries, // we make the system to allow implicitly converting String type as other primitive types. - case (_: StringType, a @ (_: AtomicType | NumericType | DecimalType | AnyTimestampType)) => - Some(a.defaultConcreteType) + case (_: StringType, a @ (_: AtomicType | NumericType | DecimalType | AnyTimestampType | + AnyTimeType)) => Some(a.defaultConcreteType) case (ArrayType(fromType, _), AbstractArrayType(toType)) => implicitCast(fromType, toType).map(ArrayType(_, true)) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala index ce387ef397aca..53de166e69edf 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala @@ -228,6 +228,7 @@ object TypeCoercion extends TypeCoercionBase { case (_: StringType, target: NumericType) => target case (_: StringType, datetime: DatetimeType) => datetime case (_: StringType, AnyTimestampType) => AnyTimestampType.defaultConcreteType + case (_: StringType, AnyTimeType) => AnyTimeType.defaultConcreteType case (_: StringType, BinaryType) => BinaryType // Cast any atomic type to string except if there are strings with different collations. case (any: AtomicType, st: StringType) if !any.isInstanceOf[StringType] => st diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercionSuite.scala index fa5027ce259d5..1f415c5ede44b 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercionSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercionSuite.scala @@ -88,6 +88,7 @@ class AnsiTypeCoercionSuite extends TypeCoercionSuiteBase { shouldCast(checkedType, DecimalType, DecimalType.SYSTEM_DEFAULT) shouldCast(checkedType, NumericType, NumericType.defaultConcreteType) shouldCast(checkedType, AnyTimestampType, AnyTimestampType.defaultConcreteType) + shouldCast(checkedType, AnyTimeType, AnyTimeType.defaultConcreteType) shouldNotCast(checkedType, IntegralType) } diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala index e6a9690ad7570..c59b687dc6ed5 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala @@ -217,6 +217,15 @@ abstract class TypeCoercionSuiteBase extends AnalysisTest { shouldNotCast(checkedType, IntegralType) } + test("SPARK-56152: implicit type cast - TimeType") { + val checkedType = TimeType() + checkTypeCasting(checkedType, castableTypes = Seq(checkedType, StringType) ++ datetimeTypes) + shouldCast(checkedType, AnyTimeType, AnyTimeType.defaultConcreteType) + shouldNotCast(checkedType, DecimalType) + shouldNotCast(checkedType, NumericType) + shouldNotCast(checkedType, IntegralType) + } + test("implicit type cast between two Map types") { val sourceType = MapType(IntegerType, IntegerType, true) val castableTypes = numericTypes ++ Seq(StringType).filter(!Cast.forceNullable(IntegerType, _)) @@ -523,6 +532,7 @@ class TypeCoercionSuite extends TypeCoercionSuiteBase { shouldCast(checkedType, DecimalType, DecimalType.SYSTEM_DEFAULT) shouldCast(checkedType, NumericType, NumericType.defaultConcreteType) shouldCast(checkedType, AnyTimestampType, AnyTimestampType.defaultConcreteType) + shouldCast(checkedType, AnyTimeType, AnyTimeType.defaultConcreteType) shouldNotCast(checkedType, IntegralType) } diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/typeCoercion/native/implicitTypeCasts.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/typeCoercion/native/implicitTypeCasts.sql.out index 977b1e1459c3e..11c24c8cc3405 100644 --- a/sql/core/src/test/resources/sql-tests/analyzer-results/typeCoercion/native/implicitTypeCasts.sql.out +++ b/sql/core/src/test/resources/sql-tests/analyzer-results/typeCoercion/native/implicitTypeCasts.sql.out @@ -359,6 +359,39 @@ Project [length(cast(cast(1996-09-10 10:11:12.4 as timestamp) as string)) AS len +- OneRowRelation +-- !query +SELECT '12:00:00' = TIME'12:00:00' FROM t +-- !query analysis +Project [(cast(12:00:00 as time(6)) = 12:00:00) AS (12:00:00 = TIME '12:00:00')#x] ++- SubqueryAlias t + +- View (`t`, [1#x]) + +- Project [cast(1#x as int) AS 1#x] + +- Project [1 AS 1#x] + +- OneRowRelation + + +-- !query +SELECT '12:00:01' > TIME'12:00:00' FROM t +-- !query analysis +Project [(cast(12:00:01 as time(6)) > 12:00:00) AS (12:00:01 > TIME '12:00:00')#x] ++- SubqueryAlias t + +- View (`t`, [1#x]) + +- Project [cast(1#x as int) AS 1#x] + +- Project [1 AS 1#x] + +- OneRowRelation + + +-- !query +SELECT time_trunc('HOUR', '12:34:56') FROM t +-- !query analysis +Project [time_trunc(HOUR, cast(12:34:56 as time(6))) AS time_trunc(HOUR, 12:34:56)#x] ++- SubqueryAlias t + +- View (`t`, [1#x]) + +- Project [cast(1#x as int) AS 1#x] + +- Project [1 AS 1#x] + +- OneRowRelation + + -- !query SELECT year( '1996-01-10') FROM t -- !query analysis diff --git a/sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/implicitTypeCasts.sql b/sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/implicitTypeCasts.sql index 6de22b8b7c3de..86efa8fa338b4 100644 --- a/sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/implicitTypeCasts.sql +++ b/sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/implicitTypeCasts.sql @@ -56,6 +56,11 @@ SELECT length('four') FROM t; SELECT length(date('1996-09-10')) FROM t; SELECT length(timestamp('1996-09-10 10:11:12.4')) FROM t; +-- string to time +SELECT '12:00:00' = TIME'12:00:00' FROM t; +SELECT '12:00:01' > TIME'12:00:00' FROM t; +SELECT time_trunc('HOUR', '12:34:56') FROM t; + -- extract SELECT year( '1996-01-10') FROM t; SELECT month( '1996-01-10') FROM t; diff --git a/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/implicitTypeCasts.sql.out b/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/implicitTypeCasts.sql.out index bb75fe5991acf..f9c32cfa7fab5 100644 --- a/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/implicitTypeCasts.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/implicitTypeCasts.sql.out @@ -263,6 +263,30 @@ struct 21 +-- !query +SELECT '12:00:00' = TIME'12:00:00' FROM t +-- !query schema +struct<(12:00:00 = TIME '12:00:00'):boolean> +-- !query output +true + + +-- !query +SELECT '12:00:01' > TIME'12:00:00' FROM t +-- !query schema +struct<(12:00:01 > TIME '12:00:00'):boolean> +-- !query output +true + + +-- !query +SELECT time_trunc('HOUR', '12:34:56') FROM t +-- !query schema +struct +-- !query output +12:00:00 + + -- !query SELECT year( '1996-01-10') FROM t -- !query schema From 2910e7fdcfe0c9a15654e1296a85e062c1da306a Mon Sep 17 00:00:00 2001 From: Chao Sun Date: Thu, 14 May 2026 21:15:01 -0700 Subject: [PATCH 125/286] [SPARK-56840][SQL][FOLLOW-UP] Add a real NullIf repro test ### What changes were proposed in this pull request? Add a focused Catalyst regression test that constructs builtin `nullif` with an unresolved nested field reference while `ALWAYS_INLINE_COMMON_EXPR=true`. This reproduces the eager `left.dataType` access that motivated SPARK-56840 and guards the fixed construction path directly. ### Why are the changes needed? The original SPARK-56840 fix was merged with an end-to-end repro that also passes without the fix, so it does not prove the bug boundary. This follow-up adds a real negative/positive regression test that fails before commit `5949ab30b41` with `Invalid call to dataType on unresolved object` and passes with the fix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.NullExpressionsSuite -- -z "NullIf accepts unresolved nested fields during inlined function construction"'` - Verified the same focused test fails on pre-fix parent `9ab8e43a940` with `UnresolvedException: Invalid call to dataType on unresolved object` and passes on current `apache/master`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex Closes #55883 from sunchao/dev/chao/codex/nullif-real-repro. Authored-by: Chao Sun Signed-off-by: Chao Sun (cherry picked from commit ffa3783fdf7caf3153b8c85a8bf4e5e4150a47c4) Signed-off-by: Chao Sun --- .../expressions/NullExpressionsSuite.scala | 18 ++++++++++++++++-- .../catalyst/optimizer/OptimizerSuite.scala | 2 +- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala index bb4aed9b40021..5c19e69cdfa3d 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala @@ -20,7 +20,8 @@ package org.apache.spark.sql.catalyst.expressions import java.sql.Timestamp import org.apache.spark.{SparkFunSuite, SparkRuntimeException} -import org.apache.spark.sql.catalyst.analysis.SimpleAnalyzer +import org.apache.spark.sql.catalyst.FunctionIdentifier +import org.apache.spark.sql.catalyst.analysis.{FunctionRegistry, SimpleAnalyzer, UnresolvedAttribute} import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext import org.apache.spark.sql.catalyst.expressions.objects.AssertNotNull import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, Project} @@ -143,7 +144,7 @@ class NullExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper { assert(analyze(new Nvl(floatLit, doubleLit)).dataType == DoubleType) } - test("NullIf replacement preserves its data type before type coercion") { + test("SPARK-56840: NullIf replacement preserves its data type before type coercion") { Seq(true, false).foreach { alwaysInlineCommonExpr => withSQLConf(SQLConf.ALWAYS_INLINE_COMMON_EXPR.key -> alwaysInlineCommonExpr.toString) { val nullIf = new NullIf(Literal(1), Literal(1)) @@ -153,6 +154,19 @@ class NullExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper { } } + test( + "SPARK-56840: NullIf accepts unresolved nested fields during inlined function construction") { + withSQLConf(SQLConf.ALWAYS_INLINE_COMMON_EXPR.key -> "true") { + val nullIf = FunctionRegistry.builtin.lookupFunction( + FunctionIdentifier("nullif"), + Seq( + UnresolvedAttribute(Seq("c", "provider")), + Lower(Literal("ERROR_MULTIPLE_PROVIDERS")))) + + assert(nullIf.isInstanceOf[NullIf]) + } + } + test("AtLeastNNonNulls") { val mix = Seq(Literal("x"), Literal.create(null, StringType), diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerSuite.scala index 057e4ceaf0a01..57b9df6512b5a 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerSuite.scala @@ -335,7 +335,7 @@ class OptimizerSuite extends PlanTest { StructType(StructField("map", MapType(IntegerType, IntegerType, false), false) :: Nil)) } - test("NullIf typed null branch is replaced with a null literal") { + test("SPARK-56840: NullIf typed null branch is replaced with a null literal") { val optimizer = new SimpleTestOptimizer() { override def defaultBatches: Seq[Batch] = Batch("test", fixedPoint, From 595ecd55793939e2c737dc49cfe7ffb5b397175b Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Fri, 15 May 2026 18:03:48 +0800 Subject: [PATCH 126/286] [SPARK-56809][UI] Show SQL description and metadata on the execution detail page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? Render the SQL execution detail page header as a single-row DataTable that reuses the same column rendering as the SQL listing page. The summary at the top of `/SQL/execution/?id=N` now shows ID, Query ID, Status, Description, Submitted, Duration, Succeeded Jobs and Error Message in the same shape as the listing page. To avoid duplicating column logic, the shared helpers, the appId resolver, the API base builder and the column factory are extracted into a new `sql-table-utils.js`, sourced by both `AllExecutionsPage` and `ExecutionPage`. The detail-page mode of the column factory: 1. skips truncation and self-links, 2. renders Description / Error Message as `
` so SQL formatting is preserved,
3. wraps long or multi-line values in a native `
` disclosure so the cells start collapsed and stay compact.

Parent / Sub Execution links remain on the detail page but move under the new summary table as a small `text-muted` line, only when applicable.

### Why are the changes needed?

The SQL execution detail page currently shows an unstructured `