Refactor collector ingest input management by thll · Pull Request #25502 · Graylog2/graylog2-server

thll · 2026-03-30T16:38:34Z

Decouples the collector ingest input from the collectors config lifecycle. Previously, the input was fully managed by the config — created, started, stopped, and deleted as part of config saves. Now the input is a regular, user-manageable input that can be independently configured, started, stopped, and deleted without affecting the collectors config or collector behavior.

The collectors config page becomes the external endpoint configuration (hostname + port pushed to collectors) plus lifecycle thresholds. Input management moves to the standard inputs page, with a convenience bootstrap on first setup and status visibility on the settings page.

Collectors always receive their endpoint configuration regardless of input state. If no input is running, collectors fail to connect and retry — their persistent queues are preserved and drain when an input comes back up. This avoids data loss from transient input restarts or accidental deletions.

/nocl

- Decouple collector ingest input lifecycle from collectors config - Remove enabled toggle and inputId linkage from IngestEndpointConfig - Expose transport config fields via allowlist in CollectorIngestHttpTransport - Extract CollectorIngestInputService from CollectorsConfigResource - Add separate GET /collectors/config/inputs endpoint for input IDs - Add create_input flag to PUT /collectors/config for onboarding bootstrap - Reject input creation in cloud environments - Simplify OpAmpService supplier to non-Optional backed by getOrDefault() - DEV migration: clean http sub-object and backfill input config defaults - Rewrite CollectorsSettings UI with Formik, extract IngestEndpointStatus - Add useCollectorInputIds hook for separate input ID fetching Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Gate create-input checkbox on inputs:create + input_types:create permissions - Map backend validation error keys to Formik field names so threshold errors surface inline on the correct TimeUnitInput fields - Remove unused Section styled component - Fix stale test: heading 'HTTP' → 'Ingest Endpoint', create_input expectation true → false for configured state, mock useInputMutations - Drop transport config overrides from createInput() and migration — only bind_address and port need explicit values, rest uses generic defaults Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace handcrafted Subject mock and SecurityTestUtils setup with the declarative @WithAuthorization annotation and extension, matching the pattern from EnrollmentTokenResourceTest. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Deleted inputs are filtered out silently via isSuccess check instead of showing a toast notification via defaultOnError. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

thll · 2026-03-31T07:03:27Z

@kroepke, @bernd, I'd like to get your opinion on the following matter.

This PR contains code to handle a scenario where a collector input exists but a user can't see it because they don't have the required permissions. However, with the Reader role granting inputs:read permission, the code is practically unreachable at the moment. Or am I missing something here?

I'm inclined to remove that code, even though it may eventually become relevant, should we ever implement entity sharing for inputs. But even if that moment ever comes, the code would need re-testing because it never triggered. What do you think?

graylog2-web-interface/src/components/collectors/settings/CollectorsSettings.test.tsx

kroepke · 2026-03-31T08:28:57Z

@kroepke, @bernd, I'd like to get your opinion on the following matter.

This PR contains code to handle a scenario where a collector input exists but a user can't see it because they don't have the required permissions. However, with the Reader role granting inputs:read permission, the code is practically unreachable at the moment. Or am I missing something here?

I'm inclined to remove that code, even though it may eventually become relevant, should we ever implement entity sharing for inputs. But even if that moment ever comes, the code would need re-testing because it never triggered. What do you think?

There are a few places in the code that still have generic INPUTS_READ permission checks without any input id. Those would also break if we added input sharing. Perhaps for clarity we should do the permission check, even if it is unreachable at the moment, so that we have a better chance of seeing it if we ever change the behavior?
Adding a small comment to that effect is probably the nicest way of leaving a breadcrumb, as well :)

thll · 2026-03-31T09:35:12Z

@kroepke, @bernd, I'd like to get your opinion on the following matter.
This PR contains code to handle a scenario where a collector input exists but a user can't see it because they don't have the required permissions. However, with the Reader role granting inputs:read permission, the code is practically unreachable at the moment. Or am I missing something here?
I'm inclined to remove that code, even though it may eventually become relevant, should we ever implement entity sharing for inputs. But even if that moment ever comes, the code would need re-testing because it never triggered. What do you think?

There are a few places in the code that still have generic INPUTS_READ permission checks without any input id. Those would also break if we added input sharing. Perhaps for clarity we should do the permission check, even if it is unreachable at the moment, so that we have a better chance of seeing it if we ever change the behavior? Adding a small comment to that effect is probably the nicest way of leaving a breadcrumb, as well :)

OK, thanks for the feedback. I added some comments and left the code in.

…ingest-input-management # Conflicts: # graylog2-web-interface/src/components/collectors/settings/CollectorsSettings.test.tsx # graylog2-web-interface/src/components/collectors/settings/CollectorsSettings.tsx

graylog2-server/src/main/java/org/graylog/collectors/opamp/OpAmpService.java

kroepke · 2026-04-07T16:00:16Z

graylog2-server/src/main/java/org/graylog/collectors/rest/CollectorsConfigResource.java

                .collectorExpirationThreshold(effectiveExpiration)
                .build();

+        if (Boolean.TRUE.equals(request.createInput())) {


can createInput just be a boolean instead of the boxed version?

…ConfigRequest

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Show error toast on input detail fetch failure instead of silently treating errors as absent inputs. Extract PortMismatchAlert into its own component file. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ingest-input-management # Conflicts: # graylog2-server/src/main/java/org/graylog/collectors/CollectorsConfigService.java # graylog2-server/src/main/java/org/graylog/collectors/input/transport/CollectorIngestHttpTransport.java # graylog2-server/src/test/java/org/graylog/collectors/CollectorCaServiceTest.java # graylog2-server/src/test/java/org/graylog/collectors/opamp/auth/AgentTokenServiceTest.java

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

thll · 2026-04-10T15:14:59Z

I've added some stuff to help the user avoid the pitfall of inadvertently configuring a different port on the input than on the collector settings page.

The generic config dialog for the collectors input now shows a description explaining the purpose of the input and the relationship to the collectors settings. The help text for the port field in that dialog mentions the configured port of the collectors settings page.

The collector settings page makes it more explicit that users are configuring the external endpoint. When inputs exist with a different port, a notification box (yay!) appears that let's the user know. It's informational, and not a warning, because this is a valid scenario.

Also fixed some stale state when navigating between tabs or between input table and settings page.

kroepke

looks good to me, much nicer to use.
i've flagged some minor fully qualified imports in tests, but nothing serious.

kroepke · 2026-04-10T16:15:03Z

graylog2-server/src/test/java/org/graylog/collectors/CollectorIngestInputServiceTest.java

+        final var input1 = mock(org.graylog2.inputs.Input.class);
+        final var input2 = mock(org.graylog2.inputs.Input.class);


should be an import

kroepke · 2026-04-10T16:15:33Z

graylog2-server/src/test/java/org/graylog/collectors/CollectorIngestInputServiceTest.java

+        when(messageInput.asMap()).thenReturn(Map.of());
+        when(messageInputFactory.create(any(InputCreateRequest.class), anyString(), isNull(), anyBoolean()))
+                .thenReturn(messageInput);
+        final var input = mock(org.graylog2.inputs.Input.class);


kroepke · 2026-04-10T16:16:00Z

...test/java/org/graylog/collectors/input/transport/CollectorIngestHttpTransportConfigTest.java

+    @Test
+    void portDescriptionIncludesConfiguredPort() {
+        final var config = CollectorsConfig.builder()
+                .http(new org.graylog.collectors.IngestEndpointConfig("host", 14401))


import as well

thll and others added 6 commits March 30, 2026 17:18

Rename status section to Ingest Inputs, use full-width layout

528d258

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Suppress error toast for stale input IDs in IngestEndpointStatus

39852ab

Deleted inputs are filtered out silently via isSuccess check instead of showing a toast notification via defaultOnError. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Revert unintended yarn.lock changes

b3dd14b

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

kroepke reviewed Mar 31, 2026

View reviewed changes

graylog2-web-interface/src/components/collectors/settings/CollectorsSettings.test.tsx Outdated Show resolved Hide resolved

thll added 4 commits March 31, 2026 10:33

Bring back useful comment

ecd47f5

Add explanatory comment about input id filtering

74f3cbd

Remove unnecessary type assertion from useInputsStates mock

062a852

Merge branch 'master' into refactor/collector-ingest-input-management

2910bac

thll marked this pull request as ready for review March 31, 2026 09:35

thll requested review from bernd and kroepke March 31, 2026 09:35

Merge remote-tracking branch 'origin/master' into refactor/collector-…

b8a96fa

…ingest-input-management # Conflicts: # graylog2-web-interface/src/components/collectors/settings/CollectorsSettings.test.tsx # graylog2-web-interface/src/components/collectors/settings/CollectorsSettings.tsx

kroepke requested changes Apr 7, 2026

View reviewed changes

thll and others added 11 commits April 9, 2026 11:26

Merge branch 'master' into refactor/collector-ingest-input-management

4f2645c

Refactor createInput parameter to use primitive boolean in Collectors…

340ea42

…ConfigRequest

feat: add static description to CollectorIngestHttpInput descriptor

d343fa5

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: dynamic port field description showing collectors config port

d159411

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: improve ingest endpoint help text and page subtitle

b914b46

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

refactor: extract useCollectorInputDetails shared hook

b7392c9

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: show port mismatch info alert on collectors settings page

9654e3a

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: refetch input data on window focus and sort mismatched ports

8bd3e65

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: lint errors in useCollectorInputDetails and PortMismatchAlert

c938813

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Clarify hostname and port being external

e25ad96

thll and others added 4 commits April 10, 2026 11:58

Clarify hostname and port being external

c008fcd

Merge branch 'master' into refactor/collector-ingest-input-management

aebb649

fix: silence 404 errors for deleted inputs in detail fetcher

40adff0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

thll requested a review from kroepke April 10, 2026 15:15

kroepke approved these changes Apr 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor collector ingest input management#25502

Refactor collector ingest input management#25502
thll wants to merge 26 commits intomasterfrom
refactor/collector-ingest-input-management

thll commented Mar 30, 2026 •

edited

Loading

Uh oh!

thll commented Mar 31, 2026

Uh oh!

Uh oh!

kroepke commented Mar 31, 2026

Uh oh!

thll commented Mar 31, 2026

Uh oh!

Uh oh!

Uh oh!

kroepke Apr 7, 2026

Uh oh!

thll commented Apr 10, 2026

Uh oh!

kroepke left a comment

Uh oh!

kroepke Apr 10, 2026

Uh oh!

kroepke Apr 10, 2026

Uh oh!

kroepke Apr 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		final var input1 = mock(org.graylog2.inputs.Input.class);
		final var input2 = mock(org.graylog2.inputs.Input.class);

Conversation

thll commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thll commented Mar 31, 2026

Uh oh!

Uh oh!

kroepke commented Mar 31, 2026

Uh oh!

thll commented Mar 31, 2026

Uh oh!

Uh oh!

Uh oh!

kroepke Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

thll commented Apr 10, 2026

Uh oh!

kroepke left a comment

Choose a reason for hiding this comment

Uh oh!

kroepke Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

kroepke Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

kroepke Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

thll commented Mar 30, 2026 •

edited

Loading