Context cancel doesn't work

[curtisr7]

I have a long running dataframe.Collect(ctx) operation and if cancel the context the job continues to run.

Is there some way that I can programatically kill the job? I see that I can get a PlanID from the dataframe, but I don't see a way to do anything with it.

[grundprinzip]

The server-side query will time-out by default after 5 minutes without the client connecting back. In addition with Spark 4 there are now a bunch more ways to cancel queries:

1. you can use tags to tag a session and then interrupt by tag see https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.SparkSession.interruptTag.html however, this isn't yet fully implemented as the public APIs in the Go DataFrame and session.

2. Spark 4 supports as well cancelling by operation ID - https://github.com/apache/spark/blob/master/sql/connect/common/src/main/protobuf/spark/connect/base.proto#L775-L786 however, as well this isn't implemented in the client yet.

3. The client should properly deal with client side cancelling of the request and cancel the query appropriately.

Generally, this seems like some work that would be great for someone to take and I'm happy to advise and review.