feat: Add Supervisor UI component to view timeline of rules applied by a cascading reindexing supervisor#19051
feat: Add Supervisor UI component to view timeline of rules applied by a cascading reindexing supervisor#19051capistrant wants to merge 29 commits into
Conversation
github-actions
Bot
added
Area - Documentation
Area - Web Console
Area - Ingestion
labels
capistrant
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
kfaraz
left a comment
kfaraz
left a comment
There was a problem hiding this comment.
Pretty cool!
Left some minor suggestions.
Will try this out locally.
Would also be nice to get a review from @vogievetsky on the web-console stuff.
| private final AppliedSkipOffset applied; | ||
| private final NotAppliedSkipOffset notApplied; |
There was a problem hiding this comment.
Nit: These two classes feel a little unnecessary. We could have just have the actual fields in the class SkipOffsetInfo directly: type, period, a boolean for isApplied, a nullable effectiveEndTime mutually exclusive with a nullable errorMessage.
| "tuning-7d", | ||
| null, | ||
| Period.days(7), | ||
| new UserCompactionTaskQueryTuningConfig( |
There was a problem hiding this comment.
Didn't we add a builder for this class in the previous PR?
| /** | ||
| * Test validation error when granularity timeline is invalid | ||
| */ |
There was a problem hiding this comment.
Not really needed since the test name captures this already.
| /** | ||
| * Test that skipOffsetFromNow correctly skips intervals and populates skipOffset.applied | ||
| */ |
There was a problem hiding this comment.
Not really needed as the test name already captures this.
| // Older data (P90D) has DAY granularity, newer data (P30D) has HOUR granularity | ||
| // This means as we move from past to present, granularity gets finer (valid) | ||
| // But then if we add MONTH for recent data, it becomes coarser (invalid) | ||
| ReindexingRuleProvider provider = InlineReindexingRuleProvider.builder() |
There was a problem hiding this comment.
Style tip: Move the .builder() to the next line to make the whole thing feel less crowded.
| ReindexingRuleProvider provider = InlineReindexingRuleProvider.builder() | |
| ReindexingRuleProvider provider = InlineReindexingRuleProvider | |
| .builder() |
| * @param referenceTime the reference time to use for computing rule periods (typically DateTime.now()) | ||
| * @return a view of the reindexing timeline with intervals and their configs | ||
| */ | ||
| public ReindexingTimelineView getReindexingTimelineView(DateTime referenceTime) |
There was a problem hiding this comment.
I feel like this method should not live here as it kind of pollutes the template definition.
Either have the timeline be built using a static method like ReindexingTimelineView.fromTemplate(template, referenceTime) or have a separate builder class altogether.
There was a problem hiding this comment.
hmm. I wonder about how we'd manage to re-use a lot of the underlying private implementation details that are quite convoluted (unfortunately). Like creating the base timeline from segment gran rules and what not is all shared between the timeline creation and the task creation. I agree that it feels a little weird here, but I didn't know how else to approach it without either duping a lot of internal logic or exposing it to both somehow. will keep thinking though
kfaraz
changed the title
kfaraz
changed the title
|
@capistrant , I took the liberty to slightly update the title for clarity. Hope it seems less ambiguous now. |
kfaraz
changed the title
|
Was trying out the embedded test in this PR to check out the timeline. @capistrant , in the snapshot below, what does the "4 rules applied" mean? Maybe we should include an info popup for each of the chips. 
|
Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
4 rules applied means that the config created for the interval is made up of 4 different reindexing rules. In this case based on the test I assume you are running, it is
You could confirm by clicking the view raw rules button. I think this is a UX issue. Hand up, I don't have any experience curating a good UX :). If I had to guess what UX decision is best here, it is to remove the As for deletion rules, N deletion rules can apply to any given interval. They accumulate (unlike other rule types). Every deletion rule that applies a search interval will be combined together.
Any interval where both of those apply would have 2 deletion rules. The chips are intended to give very quick reference to what is in the config for the given search interval. They are not strictly tied to rules even. In my pictures you see metrics and dims in separate chips, but both belong to the same rule. So maybe the deletion rules chip should actually be like 2 deletion filters or 2 deletion clauses. I didn't know how valuable chips would be for tuningConfig and ioConfig so those are not surfaced, likely leading to your initial confusion. You'd have to view the config payload or the raw rules to see them. |
I made some minor UX updates that I think help clear up some of your initial confusion. I updated the description images to reflect (2nd and 3rd image)
|
| function getPartitionType(tuningConfig: Record<string, unknown> | undefined): string | undefined { | ||
| if (!tuningConfig) return undefined; | ||
| const partitionsSpec = tuningConfig.partitionsSpec; | ||
| if (partitionsSpec && typeof partitionsSpec === 'object' && partitionsSpec !== null) { |
Check warning
Code scanning / CodeQL
Comparison between inconvertible types Warning
|
@capistrant , is this PR ready for another round of review or should we hold off until the refactor PR #19106 is reviewed and merged? |
Can probably hold off for now until the rule refactor is done |
capistrant
changed the title
| searchInterval, | ||
| referenceTime, | ||
| searchIntervals, | ||
| null |
There was a problem hiding this comment.
P2 Timeline preview drops template tuningConfig
getReindexingTimelineView builds each interval config with a ReindexingConfigBuilder whose baseTuningConfig is null, while createCompactionJobs passes the template's tuningConfig into the same builder. Any cascading supervisor with static tuning settings such as maxRowsInMemory, splitHintSpec, maxNumConcurrentSubTasks, indexSpecForIntermediatePersists, or retry/chat settings will have those settings in the actual compaction jobs but not in the new reindexingTimeline API response or downloaded UI config. This makes the preview/API incompatible with the real effective config. Pass tuningConfig here as createCompactionJobs does, and add a test that a non-partitionsSpec tuning field survives in the timeline view.
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
Reviewed 16 of 16 changed files.
This is an automated review by Codex GPT-5.5
| Interval searchInterval = intervalInfo.getInterval(); | ||
|
|
||
| // Check if interval extends past skip offset | ||
| if (intervalEndsAfter(searchInterval, effectiveEndTime)) { |
There was a problem hiding this comment.
[P2] Timeline marks partially compacted intervals as skipped
createCompactionJobs now truncates an interval that crosses the skip-offset boundary and still creates jobs for the older aligned subrange, but getReindexingTimelineView still returns ruleCount 0/config null for any interval whose end is after effectiveEndTime. For skipOffsetFromNow, and for the console's skipOffsetFromLatest preview after querying maxTime, the UI will show an interval as fully skipped even though the supervisor will compact part of it. Please mirror the truncation/splitting behavior in the timeline view so the preview matches actual job generation.
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 1 |
| Total | 2 |
Reviewed 16 of 16 changed files.
This is an automated review by Codex GPT-5.5
| queryType: 'timeBoundary', | ||
| dataSource: timelineData.dataSource, | ||
| }; | ||
| const resp = await Api.instance.post('/druid/v2', query); |
There was a problem hiding this comment.
[P2] Compute skipOffsetFromLatest from the supervisor timeline
The Query latest timestamp path bases the preview boundary on a broker timeBoundary maxTime, then subtracts the period in the browser. The actual cascading supervisor path applies skipOffsetFromLatest to the segment timeline end and then truncates to an interval granularity-aligned boundary, so row maxTime can be earlier than the segment end and straddling intervals are handled differently. For normal DAY/HOUR segments this can show intervals as skipped or active at the wrong boundary. Please derive the preview from segment metadata/server-side timeline logic, or mirror the same segment-end and granularity-alignment rules.
| active: activeTab === 'history', | ||
| onClick: () => setActiveTab('history'), | ||
| }, | ||
| ...(isCompactionSupervisor |
There was a problem hiding this comment.
[P3] Hide the timeline tab for non-cascading compaction supervisors
This adds the Timeline tab for every supervisor ID that starts with autocompact__, but the new endpoint returns 400 unless the compaction supervisor template is CascadingReindexingTemplate. Existing inline-config compaction supervisors also use autocompact__ IDs, so standard compaction supervisors now expose a tab that only shows an error. Gate the tab on the loaded supervisor spec/template type, or render a deliberate unavailable state instead of adding the tab solely from the ID prefix.
4 participants
Description
tied to the effort in #19092 to productionalize this new feature
New Supervisor API (cascading reindexing supervisors only)
This generates a timeline of search intervals for reindexing with the effective sets of rules that are used to create the underlying inline compaction configs for the different intervals. This is the business end of the console UI described below
Console visualization of reindexing timeline (cascading reindexing supervisors only)
For cascading compaction supervisors you can get a timeline view of how rules from the rule provider(s) will apply to your datasource. You can view how skip offsets (if they are configured) will change the timeline too. This visualization is a great help for operators in understanding in how their underlying rules will be applied across the datasource history. I will add some screen shots for reference
Refactor
CompactionSupervisorTestEmbedded TestsSplit out the cascading template tests to their own class from the inline compaction supervisor tests. Sharing one file was getting convoluted. They share a base class for what makes sense.
Try it Yourself
If you build this branch locally and start up a cluster. post the following spec example to supervisors api and you can view a prospective timeline plus play with the config if you want to see how it can change. note that you do not have to pre-ingest the wikipedia data! this supervisor will be suspended and has no reliance on underlying data (caveat is that if you try to do skipOffsetFromLatest and click the dynamic button to calculate, you would need to have the data ingested)
Click to expand code
{ "type": "autocompact", "spec": { "type": "reindexCascade", "dataSource": "wikipedia", "taskPriority": 25, "inputSegmentSizeBytes": 100000000000000, "ruleProvider": { "type": "inline", "deletionRules": [ { "id": "delete_japan", "description": null, "olderThan": "P90D", "deleteWhere": { "type": "equals", "column": "countryName", "matchValueType": "STRING", "matchValue": "Japan" }, "virtualColumns": null } ], "partitioningRules": [], "indexSpecRules": [], "dataSchemaRules": [ { "id": "dataSchema-1", "description": null, "olderThan": "P30D", "dimensionsSpec": { "dimensions": null }, "metricsSpec": [], "queryGranularity": "HOUR", "rollup": true, "projections": [] } ] }, "taskContext": { "useConcurrentLocks": true, "maxNumTasks": 2 }, "skipOffsetFromLatest": null, "skipOffsetFromNow": null, "defaultSegmentGranularity": "DAY", "defaultPartitionsSpec": { "type": "range", "targetRowsPerSegment": 5, "maxRowsPerSegment": null, "partitionDimensions": [ "city_and_country", "isRobot" ], "assumeGrouped": false }, "defaultPartitioningVirtualColumns": [ { "type": "expression", "name": "city_and_country", "expression": "concat(cityName, ':', countryName)", "outputType": "STRING" } ], "tuningConfig": null }, "suspended": true }Release note
Key changed/added classes in this PR
CascadingReindexingTemplateReindexingTimelineViewSupervisorResourceweb-console/src/components/reindexing-timeline/reindexing-timeline.scssThis PR has: