Summary query data pannel freeze when the number of queries count is high

[MonkeyCanCode]

Describe the bug
When a spark cluster has high number of queries ran (e.g. 800), the summary page for the query data panel is freezing all the time. Looking at the network tab, this appear to be data fetching from http://localhost:4040/api/v1/applications/local-1767169191734/stages?withSummaries=false&quantiles=0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0 which can take couple seconds (but this is async, so it is okay). However, the data processing with the large json payload from this API call via calculateStagesStore seems to be causing issues (it can take 4-5 seconds which then locked up main thread it seems).

Environemnt
spark verison: 4.x
platform: standalone (local spark connect)

To Reproduce

1. Start local spark connect

sbin/start-connect-server.sh

2. Connect via a local client and run sample query for 800 times (as the fetch is up to 1000 queries, so this is a valid use case)

from pyspark.sql import SparkSession
from pyspark.sql.functions import avg

spark = SparkSession.builder.remote("sc://localhost/").getOrCreate()

df = spark.createDataFrame(
    [
        ("sue", 32),
        ("li", 3),
        ("bob", 75),
        ("heo", 13),
    ],
    ["first_name", "age"],
)

for i in range(800):
    df.select(avg("age")).show()

3. Open http://localhost:4040/dataflint/#/summary and scroll on the queries data panel and it should freeze every couple seconds

Expected behavior
It should not freeze during scrolling on the data panel.

Screenshots
N/A

Additional context
N/A

[menishmueli]

Hi @MonkeyCanCode , I tool a look and manage to recreate the problem.

The first walkaround I can offer is to open large application logs in Spark History Server - DataFlint does not refresh in this state so there won't be freezes when the state is being recompute.

I'm checked and the issue is not the network or the json - is some recalculations that happen regardless of if the stages updated or not. Working on trying to mitigate this :) but it's not trivial fix.

[MonkeyCanCode]

Hi @MonkeyCanCode , I tool a look and manage to recreate the problem.

The first walkaround I can offer is to open large application logs in Spark History Server - DataFlint does not refresh in this state so there won't be freezes when the state is being recompute.

I'm checked and the issue is not the network or the json - is some recalculations that happen regardless of if the stages updated or not. Working on trying to mitigate this :) but it's not trivial fix.

I am not sure if using spark history server to get this will be viable as that log size can be huge for long running application (plus, we probably can't really guarantee everyone has spark history server infra as well). And yes, the issue is recompute for sure as the pulling is async (this will be 2-3s but it is async) but the recompute is the that is blocking main thread when I checked earlier. Wondering if we should use web worker for it as CPU intensive compute doesn't block main thread.

[menishmueli]

@MonkeyCanCode when you try to use DataFlint, do you have running queries or the cluster is innactive?
I'm adding logic so if nothing changed in terms of SQL and stages, DataFlint skips a lot computation it does every second. WDYT?

[MonkeyCanCode]

@MonkeyCanCode when you try to use DataFlint, do you have running queries or the cluster is innactive?
I'm adding logic so if nothing changed in terms of SQL and stages, DataFlint skips a lot computation it does every second. WDYT?

For my setup (spark connect), the cluster will be always on and client (spark client) will run ingestion on batch with scheduled time (e.g. every x mins, this can be as low as 1 min).

[menishmueli]

Yes we could, will be added as config for next version.
For now we added some updates in version 0.8.1, please give it a try :)

[kullanici0606]

We also experience similar issue for our structured streaming job, we are currently on version 0.8.3. If you need any additional data, I can provide it.