feat(stack): allow controller metrics scraping#95
Conversation
![kubexautomation[bot]](https://avatars.githubusercontent.com/u/44006299?s=60&v=4){kind=small}
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: This is a focused, low-risk single-line change that adds the kubex-automation-engine metrics service to the Prometheus endpointslice scrape allowlist. The intent is clear, the testing evidence is solid, and the change is confined to a single value. One minor regex hygiene issue and a missing chart version bump are worth addressing.
Risk level: Low
Critical issues:
- None
Major issues:
- None
Minor issues:
-
Redundant trailing
.*inside the new alternative (Location:charts/kubex-automation-stack/values.yaml, changed line — the newregexvalue)The new alternative is
.*kubex-automation-engine.*metrics-service.*, but the outer regex already has a trailing.*applied to the entire group(…).*. The extra.*at the end of the inner alternative is redundant, adds noise, and makes the regex slightly harder to audit.Suggested fix — make the inner alternative consistent with its siblings:
regex: '((kubex|densify)-(kube-state-metrics|prometheus-node-exporter|ephemeral-storage-collector)|.*dcgm|k8s-ephemeral-storage-metrics|.*kubex-automation-engine.*metrics-service).*'
-
Over-broad leading wildcard (Confidence: Low — may be intentional)
The leading
.*in.*kubex-automation-engine.*metrics-servicemeans any endpointslice name containingkubex-automation-engineand latermetrics-servicewill be scraped, regardless of prefix. Contrast this with the other fixed-prefix alternatives (kubex-kube-state-metrics,densify-prometheus-node-exporter, etc.). If the Helm release name is alwayscontroller(as shown in the test output), a more anchored pattern such ascontroller-kubex-automation-engine-metrics-servicewould be less permissive. This is low-confidence feedback — the.*may be intentional to handle arbitrary release name prefixes. -
No chart version bump (Confidence: Medium)
The
charts/kubex-automation-stack/Chart.yamldoes not appear to have been updated in this PR. Adding a new scrape target tovalues.yamlis a functional behaviour change for users of the bundled Prometheus stack, so a patch-version bump (e.g.,0.x.y → 0.x.(y+1)) and aCHANGELOG/NOTES.txtentry would follow standard Helm chart conventions and aid in upgradability tracking.
DRY improvement opportunities:
- None in scope for this change.
Suggested next steps:
- Remove the redundant trailing
.*from inside the new regex alternative. - Consider whether the leading
.*in.*kubex-automation-engine.*metrics-serviceshould be anchored more tightly (e.g., use a known release-name prefix), or document in a comment why the wildcard prefix is required. - Bump the
kubex-automation-stackchart patch version inChart.yamlto reflect the behavioural change in scrape configuration.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: This is a well-scoped, minimal change that adds the kubex-automation-engine controller metrics service to the Prometheus endpointslice scrape allowlist and correctly bumps the chart version (1.0.8 → 1.0.9). However, the change introduces a correctness risk around TLS handling and a likely silent data-loss issue due to the existing metric_relabel_configs allowlist that was not updated to admit the new service's metrics.
Risk level: Medium
Critical issues:
- None
Major issues:
scheme/ TLS mismatch (values.yamlline 219)
The test output shows the controller metrics run on port8443(HTTPS). Thekubernetes-service-endpointslicejob usesscheme: httpunless theServicecarriesprometheus.io/scheme: https. If that annotation is absent, all scrapes will silently fail (up == 0). The PR should either confirm the annotation is present in the engine chart or add a dedicated HTTPS-capable scrape job.
Minor issues:
-
metric_relabel_configsallowlist drops all controller metrics (values.yamlline 219)
The existingmetric_relabel_configsin thekubernetes-service-endpointslicejob is a tightly-scoped allowlist for kube-state-metrics, node-exporter, DCGM, and ephemeral storage metrics.kubex-automation-enginemetrics (controller-runtime, custom business metrics) will be scraped and then entirely dropped. If the intent is to expose controller metrics to users of the stack, the allowlist must be extended with the relevant metric name patterns — or a separate job without the allowlist should be used. -
Over-broad leading
.*in the regex alternative (values.yamlline 219)
The pattern.*kubex-automation-engine.*metrics-servicecan match unintended endpointslices. All other alternatives in the group use fixed prefixes. If the Helm release name is alwayscontroller, consider anchoring tocontroller-kubex-automation-engine-metrics-service. If arbitrary release names must be supported, document the reason with an inline comment.
DRY improvement opportunities:
- The PR description mentions "Prometheus Operator users" but no
ServiceMonitorCRD resource changes appear in the diff. If Prometheus Operator support was deferred, tracking it as a follow-up issue would be useful.
Suggested next steps:
- Confirm (or enforce via chart annotations) that the
kubex-automation-enginemetricsServicecarriesprometheus.io/scheme: https— or create a dedicated scrape job that hard-codesscheme: httpsand TLS config. - Extend the
metric_relabel_configskeep-regex to includekubex-automation-enginemetric name patterns, or isolate the controller scrape in a separate job without the restrictive allowlist. - Consider tightening the endpointslice regex to avoid inadvertent scraping of similarly-named services.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: This PR correctly extends the endpointslice scrape allowlist and updates the metric_relabel_configs keep-regex to admit controller metrics. The chart version bump (1.0.8 → 1.0.9) is present. However, two structural issues introduce correctness risk — the TLS/scheme dependency on a Service annotation that may not be set, and the introduction of very broad metric-name wildcards (go_.*, process_.*, workqueue_.*, etc.) into a shared, tightly-scoped allowlist.
Risk level: Medium
Critical issues:
- None
Major issues:
-
TLS scheme dependency on Service annotation (
values.yamlline 219)
Thekubernetes-service-endpointslicejob only uses HTTPS when the Service carriesprometheus.io/scheme: https. The controller metrics run on port8443(HTTPS). If the engine chart does not set this annotation by default, all scrapes will silently fail (up == 0) in user environments even though the PR's local Kind test passed. A dedicated job with hard-codedscheme: https(like thekubernetes-apiserversjob) removes this fragile annotation dependency. -
Broad metric-name wildcards in a shared allowlist (
values.yamlline 228)
The new patternsgo_.*,process_.*,workqueue_.*,controller_runtime_.*, andrest_client_.*apply to all targets scraped bykubernetes-service-endpointslice— not just the controller metrics service. Any other Go-based workload added to the endpointslice allowlist in the future will have its Go runtime metrics retained without an explicit choice. A dedicated scrape job for the controller metrics keeps the allowlist intent intact and avoids this cross-contamination.
Minor issues:
- Over-broad leading
.*in endpointslice regex (values.yamlline 219)
The pattern.*kubex-automation-engine.*metrics-serviceis more permissive than all peer alternatives. Tightening to.*-kubex-automation-engine-metrics-service(or anchoring to the known release prefixcontroller-) reduces inadvertent matches. If the wildcard is intentional, a short inline comment stating why would prevent future regressions.
DRY improvement opportunities:
- Both the endpointslice filter and the metric allowlist would be cleaner and safer if the controller metrics used a dedicated scrape job rather than being co-located in
kubernetes-service-endpointslice. This is a design suggestion, not a blocker — but it resolves issues #1, #2, and #3 above in one move.
Suggested next steps:
- Confirm or enforce that the
kubex-automation-enginemetrics Service hasprometheus.io/scheme: https. If that cannot be guaranteed, create a dedicatedkubernetes-controller-metricsscrape job withscheme: httpsand in-cluster TLS config. - Move the controller-specific metric-name patterns (
controller_runtime_.*,go_.*,process_.*,workqueue_.*,rest_client_.*) out of the sharedkubernetes-service-endpointsliceallowlist — either into a dedicated job or guarded by a service-label condition. - Tighten or document the leading
.*in the endpointslice keep-regex.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: This PR correctly introduces a dedicated scrape job (kubex-automation-engine-metrics-endpointslice) for the controller metrics service rather than co-opting the shared kubernetes-service-endpointslice job — a sound architectural decision that avoids polluting the shared metric_relabel_configs allowlist. The new job also correctly omits metric_relabel_configs entirely, so all controller metrics pass through unfiltered. The main risk is that TLS and authentication are not hard-coded in the new job, making it fragile when Service annotations are absent.
Risk level: Medium
Critical issues:
- None
Major issues:
-
Missing TLS config and bearer token (
values.yaml, line 232 — new job declaration)
The new job relies entirely on theprometheus.io/schemeannotation to select HTTPS, and provides nobearer_token_file. The controller metrics endpoint runs on port8443(HTTPS, likely RBAC-protected). If the Service lacksprometheus.io/scheme: https, the job defaults to HTTP and scrapes fail silently. If the endpoint requires a bearer token (standard for controller-manager metrics), it will return401regardless of TLS. Thekubernetes-apiserversjob in the same file shows the correct pattern: hard-codescheme: https,tls_config, andbearer_token_file. -
Verbatim copy-paste of 10
relabel_configsrules (values.yaml, lines 237–268)
The entirerelabel_configsblock is duplicated fromkubernetes-service-endpointslicewith only the keep-filter regex changed. Any future maintenance change to the shared relabelling logic must be applied in two places with no mechanical enforcement. A Helm named template in_helpers.tplor a YAML anchor would eliminate this duplication.
Minor issues:
-
Over-broad and redundant regex (
values.yaml, line 260)
'.*kubex-automation-engine.*metrics-service.*'has a leading.*that permits accidental matches in multi-tenant clusters, and a trailing.*that is functionally redundant. All peer alternatives use anchored prefixes. Consider tightening to'controller-kubex-automation-engine-metrics-service'(or document why a wildcard prefix is required). -
Version skip: 1.0.8 → 1.0.10 (
Chart.yaml, line 4)
Skipping1.0.9creates a gap in the release history. If1.0.9was never published this is a safe sequential bump to1.0.9. If it was published separately, document the reason for the skip in the PR description.
DRY improvement opportunities:
- The shared
relabel_configslogic (scheme override, path override, address rewrite, label-map, namespace/service/node label extraction) is now duplicated across two jobs. Extracting it into a Helm named template is the highest-value refactoring opportunity in this file.
Suggested next steps:
- Hard-code
scheme: https,tls_config.ca_file, andbearer_token_filein the new job (mirroring thekubernetes-apiserversjob), or confirm and document that thekubex-automation-enginemetrics Service always carriesprometheus.io/scheme: https. - Extract the shared
relabel_configsblock into a Helm named template to eliminate the copy-paste duplication. - Tighten the endpointslice keep-regex or add an inline comment explaining the wildcard prefix.
- Reconcile the version skip (1.0.8 → 1.0.10) with an explanation or correct to 1.0.9.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: This PR introduces a well-structured dedicated scrape job for the kubex-automation-engine controller metrics, correctly avoiding co-option of the shared kubernetes-service-endpointslice job and its tight metric_relabel_configs allowlist. The architecture is sound. However, one correctness risk remains around TLS/auth configuration, and there are minor issues with the endpointslice regex and the version bump.
Risk level: Medium
Critical issues:
- None
Major issues:
scheme: httpvs observed HTTPS port + missing bearer token (values.yamlline 232)
The inline comment says the controller exposes metrics over HTTP on port 8080, but the PR's own test output shows the scraped instance at10.244.0.16:8443— an HTTPS port.scheme: httpcombined with nobearer_token_filemeans that in any environment where the endpoint is actually HTTPS/RBAC-protected, every scrape will fail silently with a TLS error or a401. Thekubernetes-apiserversjob in the same file shows the correct pattern: hard-codedscheme: https,tls_config.ca_file, andbearer_token_file. This should be reconciled before merging.
Minor issues:
-
Over-broad and partially redundant endpointslice keep-regex (
values.yamlline 258)
'.*kubex-automation-engine.*metrics-service.*'has a leading.*that allows accidental matches in multi-tenant clusters (all peer alternatives use fixed prefixes), and a trailing.*that is redundant in Prometheus relabelaction: keepregex matching. Consider anchoring to'controller-kubex-automation-engine-metrics-service'or documenting why a wildcard prefix is required. -
Version skip: 1.0.8 → 1.0.11 (
Chart.yamlline 4)
The chart version jumps two patch levels with no explanation. If 1.0.9 and 1.0.10 were never published this should be 1.0.9. If they were published, the PR description should explain the skip.
DRY improvement opportunities:
- The 10-rule
relabel_configsblock is copy-pasted verbatim from thekubernetes-service-endpointslicejob. Extracting the shared rules into a Helm named template (_helpers.tpl) would eliminate the duplication and ensure the two jobs stay in sync during future maintenance.
Suggested next steps:
- Reconcile
scheme: httpwith the observed HTTPS/8443 endpoint. If HTTPS, addscheme: https,tls_config.ca_file, andbearer_token_file(mirror thekubernetes-apiserversjob). Update the inline comment accordingly. - Tighten the endpointslice keep-regex or add an inline comment explaining why a wildcard prefix is required.
- Correct the chart version to
1.0.9(or document the skip to1.0.11). - (Non-blocking) Extract shared
relabel_configsinto a Helm named template to eliminate copy-paste drift.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: This PR adds a new dedicated Prometheus scrape job for the kubex-automation-engine controller metrics endpoint. The approach is reasonable, but there is a critical factual contradiction in the code (HTTP vs HTTPS) that must be resolved before merging, and the absence of metric_relabel_configs represents an intentional (but undocumented) architectural departure from the rest of the file.
Risk level: Medium
Critical / Major issues:
-
HTTP/HTTPS contradiction (values.yaml:232): The new job hard-codes
scheme: httpand an inline comment says the port is8080 (HTTP), but the PR's own verified test output shows the instance at10.244.0.16:8443— a standard HTTPS port. If the endpoint is HTTPS, every scrape will silently fail (up == 0) on a fresh install. Additionally, if the endpoint is RBAC-protected (common for controller-manager metrics), abearer_token_fileis required — see the existingkubernetes-apiserversjob for the established pattern in this file. This must be clarified before merge. -
No
metric_relabel_configs(values.yaml:258 area): All other scrape jobs in this file use a tightly-scoped allowlist to control stored metric cardinality. This new job stores everything the controller emits (699 samples in test). That may be intentional for full controller observability, but it should be explicitly documented — either with a comment or by adding a scoped allowlist.
Minor issues:
-
Overly broad keep-regex (values.yaml:261):
'.*kubex-automation-engine.*metrics-service.*'uses open-ended wildcards on both sides. All peer patterns in this file use fixed prefixes. The PR test output shows the name iscontroller-kubex-automation-engine-metrics-service; consider anchoring with.+-kubex-automation-engine-metrics-serviceor documenting the intent of the broad pattern. The trailing.*is also redundant. -
Chart version (Chart.yaml:4): The version jumps from
1.0.8to1.0.9in this PR, which is fine if1.0.9has not been published. If other changes are already staged for1.0.9or it has already been released, coordinate to avoid a version collision.
DRY improvement opportunity:
- Copy-pasted
relabel_configs(values.yaml:236–267): The 9-rule relabel block is a near-verbatim copy of the block inkubernetes-service-endpointslice. As a Helm chart, extracting shared rules into a_helpers.tplnamed template would eliminate future drift. Non-blocking, but worth tracking as a follow-up.
Suggested next steps:
- Clarify and fix the HTTP vs HTTPS discrepancy — confirm the actual protocol/port and update
scheme,tls_config, andbearer_token_fileaccordingly. - Add a
metric_relabel_configsblock (or an explicit comment) documenting which controller metrics are intentionally stored. - Tighten the endpointslice keep-regex to prevent unintended multi-tenant matches.
- (Follow-up) Extract shared relabel rules into a
_helpers.tpltemplate.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: The PR makes a good structural improvement by introducing YAML anchors (&/*) to DRY up the shared relabel_configs between the two scrape jobs — this directly addresses duplication that was flagged in earlier review rounds. However, the refactoring introduced two behavioral regressions in the existing kubernetes-service-endpointslice job, and the new controller metrics job has an unresolved HTTP/HTTPS ambiguity that could cause silent scrape failures.
Risk level: Medium
Critical issues
None.
Major issues
-
serviceandnodelabels silently dropped fromkubernetes-service-endpointslice(regression)
Location:charts/kubex-automation-stack/values.yaml(diff context around line 219)
The two relabel rules that populated theservice(from__meta_kubernetes_service_name) andnode(from__meta_kubernetes_pod_node_name) labels were removed as part of the YAML anchor refactoring. These labels are not part of the shared anchors, so every time series scraped by the existing job (kube_*,node_*, DCGM, ephemeral storage metrics) will lose those labels after a chart upgrade. Any dashboard or alert filtering on{service="..."}or{node="..."}for those metrics will break silently. -
scheme: httphard-coded on new job, but test shows port 8443 (HTTPS)
Location:charts/kubex-automation-stack/values.yaml:231
The PR's test output showsinstance: 10.244.0.16:8443. Port 8443 is universally HTTPS. The new job neither inherits the annotation-driven scheme rule (it was excluded from the shared anchors) nor provides TLS/bearer token config. If the endpoint is HTTPS, every scrape will fail with a TLS handshake error (up == 0). Needs explicit clarification and, if HTTPS,scheme: https+tls_config+bearer_token_file. -
prometheus.io/schemeannotation rule moved to after thekeepfilter inkubernetes-service-endpointslice
Location:charts/kubex-automation-stack/values.yaml:222
The scheme-detection rule was repositioned lower in therelabel_configsblock. While functionally harmless in current Prometheus versions (since__scheme__is consumed at scrape time), the new placement buries it after thekeepfilter and reduces clarity. It now appears after namespace assignment rather than as the first visible signal that HTTPS endpoints are supported. At minimum, an inline comment should explain the reordering.
Minor issues
-
Undocumented
.+wildcard prefix in new job'skeepregex
Location:charts/kubex-automation-stack/values.yaml:248
The regex.+-kubex-automation-engine-metrics-serviceuses.+to accommodate any Helm release name prefix. All other patterns in the sibling job use fixed prefixes. Without a comment explaining the intent, future maintainers may tighten this inadvertently (or leave it too loose in multi-tenant clusters). Add an inline comment. -
No
metric_relabel_configson new job — intentional but underdocumented
Location:charts/kubex-automation-stack/values.yaml— new job block (after line 248)
The inline comment (# Store the controller-runtime, Go, process, and workqueue metrics as-is.) partially documents the intent. Consider enumerating the expected metric families (controller_runtime_, go_, process_, workqueue_) to make future audits easier and set a clear boundary for what's expected.
DRY improvement opportunities
- The YAML anchor approach (
&/*) is the right move and correctly DRYs up the shared relabel rules. No further DRY work is needed for this PR; the Helm_helpers.tplapproach suggested in earlier reviews is no longer necessary given the anchor solution.
Suggested next steps
- Confirm whether
serviceandnodelabel removal is intentional. If it is an accidental side-effect of the anchor refactoring, restore those two rules to thekubernetes-service-endpointslicejob (they should not be part of the shared anchors since the new job doesn't need them). - Resolve the HTTP vs HTTPS ambiguity for the controller metrics endpoint and update
scheme:and TLS/auth config accordingly on the new job. - Move or annotate the scheme rule in the existing job so its purpose remains obvious to readers.
- Add a comment on the
.+wildcard in the new job's keep regex explaining that it intentionally supports variable Helm release name prefixes.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: The PR introduces a well-structured dedicated scrape job for kubex-automation-engine controller metrics. The YAML anchor (&/*) approach cleanly solves the DRY problem that earlier review iterations flagged, and correctly preserves service and node labels in the existing kubernetes-service-endpointslice job via the anchors. The architecture is sound. One factual contradiction must be resolved before merging, and a missing authentication consideration should be addressed.
Risk level: Medium
Critical issues:
- None
Major issues:
-
HTTP/HTTPS contradiction —
scheme: httpvs port8443in test output (values.yamlline 240)
The job hard-codesscheme: httpand the inline comment says port8080 (HTTP), but the PR's own verified test output showsinstance: 10.244.0.16:8443— an HTTPS port. Additionally, unlike the siblingkubernetes-service-endpointslicejob, the new job'srelabel_configsdoes not include theprometheus.io/schemeannotation-override rule (intentionally excluded from the shared anchors), so there is no annotation-based escape valve. If the endpoint is HTTPS, every scrape will fail withup == 0on a fresh install, silently.
→ Must be resolved before merge. Confirm the actual protocol/port and either keepscheme: http(with an accurate comment) or switch toscheme: httpswithtls_config+bearer_token_file, mirroring thekubernetes-apiserversjob pattern. -
No
bearer_token_file(values.yamlline 240)
Controller-runtime/kubebuilder metrics endpoints are commonly RBAC-protected. Withoutbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token, Prometheus will receive401 Unauthorizedif the endpoint requires authentication. The existingkubernetes-apiserversjob in this file always sets the bearer token. If thekubex-automation-enginemetrics endpoint is explicitly unauthenticated, that should be confirmed with an inline comment; otherwise the token should be added.
Minor issues:
-
Endpointslice keep-regex may match suffix variants (
values.yamlline 260)
.+-kubex-automation-engine-metrics-serviceuses.+with implicit Prometheus full-string anchoring (Goregexp.MatchStringwith full-value matching foraction: keep). The inline comment correctly explains the variable-prefix intent. No change strictly required, but explicitly documenting that Prometheus matches the full endpointslice name string (not a substring) would remove ambiguity for future maintainers. -
No documented metric cardinality boundary for the new job (
values.yamltrailing comment on new job)
The comment# Store the controller-runtime, Go, process, and workqueue metrics as-is.correctly signals the unbounded-storage intent, but names the metric families only partially. Enumerating the full expected set (controller_runtime_*,go_*,process_*,workqueue_*,rest_client_*) and noting where to add an allowlist if cardinality becomes a concern would make this a more durable guideline.
DRY improvement opportunities:
- The YAML anchor approach is the right solution for this codebase and has been correctly implemented. No further DRY work is needed in this PR. The
_helpers.tplapproach suggested in earlier reviews is superseded.
Suggested next steps:
- Clarify and fix the
scheme/port discrepancy — confirm HTTP/8080 or HTTPS/8443, update the job config and inline comment to match reality. - Add
bearer_token_file(or explicitly document that the endpoint is unauthenticated). - (Optional) Expand the trailing comment on the new job to enumerate all expected metric families as a cardinality guardrail.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall Assessment
This PR successfully introduces a dedicated Prometheus scrape job for kubex-automation-engine controller metrics and uses YAML anchors (&/*) to eliminate the previous copy-paste duplication of relabel_config rules — a genuine improvement over the earlier approach. The chart version bump (1.0.8 → 1.0.9) is sequential and correct. The keep filter reordering (moved after service/node label rules) is functionally safe since __meta_kubernetes_endpointslice_name is a discovery meta-label available from the start of relabelling.
The one outstanding blocker is an unresolved contradiction between the inline comment (port 8080, plain HTTP) and the PR's own verified test output (instance: 10.244.0.16:8443, an HTTPS port). This ambiguity must be resolved before merge to avoid silent scrape failures (up == 0) for chart consumers.
Risk level: Medium
Critical issues
None.
Major issues
scheme: httpvs port8443contradiction (values.yamlline 240): The inline comment claims HTTP/8080 but the PR test output shows HTTPS/8443. Because the shared YAML anchors deliberately exclude theprometheus.io/schemeannotation-override rule, there is no runtime escape valve —httpis hardwired for this job. If the endpoint is actually HTTPS, every scrape on a fresh install will silently fail.- Missing
bearer_token_file(values.yamlline 241): No comment confirms the endpoint is unauthenticated. Controller-runtime operators typically RBAC-protect their/metricsport; if that is the case here, Prometheus will receive401 Unauthorized. The existingkubernetes-apiserversjob in this file setsbearer_token_fileexplicitly — the new job should either do the same or clearly document why it is not needed.
Minor issues
- Regex end-of-string anchoring (
values.yamlline 260):.+-kubex-automation-engine-metrics-servicemay match service names with unexpected suffixes (e.g.,-canary,-v2) depending on whether Prometheus applies full-string or substring matching semantics. Adding a trailing$or a confirming comment about Prometheus's implicit anchoring behaviour would eliminate the ambiguity. - Trailing comment lacks cardinality expectations (
values.yamlline 261): All other jobs in this file use strictmetric_relabel_configsallowlists. The new job intentionally omits them, but the comment should enumerate the expected metric families and note that ametric_relabel_configsblock should be added if cardinality becomes a concern.
DRY improvement opportunities
- The YAML anchor approach introduced in this PR is the right direction and correctly avoids the copy-paste problem flagged in earlier review rounds. No further DRY action is needed at this time.
Suggested next steps
- Clarify the protocol/port used by the
kubex-automation-enginemetrics Service. If it is HTTP/8080, update the PR test evidence or explain the8443instance. If it is HTTPS/8443, replacescheme: httpwithscheme: https+tls_config+bearer_token_file. - Confirm or document authentication requirements for the
/metricsendpoint in thekubex-automation-enginechart. If unauthenticated, say so explicitly in the comment. - Optionally anchor the keep-regex with
$and expand the trailing comment to enumerate expected metric families for future maintainability.
|
Follow-up on the latest review items:
Validation rerun after these follow-ups:
All GitHub review threads on this PR are now resolved. |
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment
This PR introduces a well-structured approach to reusing Prometheus relabel config blocks via YAML anchors/aliases, and the dedicated scrape job for controller metrics is the right architectural choice. However, there is a critical contradiction between the inline comment and the PR's own test evidence regarding HTTP vs HTTPS, plus a missing metric_relabel_configs allowlist that creates an open-ended cardinality risk in production.
Risk level: Medium
Critical issues
scheme: httpvs HTTPS/8443 contradiction (values.yaml:240): The PR test output shows the scrape target at port8443(HTTPS), but the inline comment andscheme: httpclaim port 8080/HTTP. One of these is wrong. If the endpoint is actually HTTPS and requires TLS + a bearer token, this job will produce zero data silently.
Major issues
- No
metric_relabel_configsallowlist (values.yaml:265): Every other job in this file gates metrics through a strict allowlist. Leaving the new job completely open storesgo_*,process_*,workqueue_*, andrest_client_*metrics unbounded — families that are high-cardinality and shared with many other in-cluster components. - Unanchored keep-regex (
values.yaml:261):.+-kubex-automation-engine-metrics-servicewill match any EndpointSlice name ending with that suffix in any namespace, potentially scraping unintended targets in multi-tenant or multi-release deployments.
Minor issues
prometheus.io/portannotation dependency (values.yaml:243): The reused*kubexEndpointsliceAddressalias relies on theprometheus.io/portannotation to set the correct port. Verify the controller metrics service carries this annotation; if not, the port is inferred from the EndpointSlice and may not match the expected 8080.- Relabel rule reordering in the existing job (
values.yaml:225): Theservice/nodelabel assignments were moved ahead of theendpointslice_namekeep-filter to enable anchor placement. Functionally safe, but worth confirming the intent is deliberate.
DRY improvement opportunities
- The YAML anchor/alias pattern used here is a good improvement for this file, but anchors in
values.yamlare YAML-native only — they do not survivehelm templaterendering for users who overridevalues.yamlexternally (e.g., via--setor a separatevalues.yamlfile). This is an inherent limitation of the approach. A Helm_helpers.tplnamed template would be more robust for cross-file reuse, but the YAML anchor approach is acceptable within a singlevalues.yaml.
Suggested next steps
- Resolve the HTTP/HTTPS contradiction by checking the actual
--metrics-bind-addressflag on the controller deployment and matchingscheme,tls_config, andbearer_token_fileaccordingly. - Add a
metric_relabel_configsallowlist scoped toautomation_controller_*and the specificcontroller_runtime_*families you need, rather than leaving the job open. - Tighten the EndpointSlice keep-regex or add a namespace filter to prevent cross-tenant scraping.
|
Fresh end-to-end validation on a new Kind cluster surfaced one real issue in the dedicated controller scrape job and confirmed the rest of the recent review concerns were either stale or intentional:
Fresh validation results after the fix:
This also means the earlier |
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: This PR cleanly adds controller metrics scraping via a new dedicated Prometheus scrape job and uses YAML anchors to reduce duplication across relabel configs — a solid improvement over copy-paste. The author has already addressed many issues from prior review rounds. A few residual concerns remain, primarily around honor_labels, a reordering in the existing job, and a missing regex anchor.
Risk level: Medium
Major issues:
-
honor_labels: truein the new scrape job (values.yaml ~line 235): Allows scraped metric labels to silently override Prometheus server-side labels (job,instance,namespace). This is unsafe for a direct controller-runtime endpoint and should be removed. See inline comment. -
keeprule moved to the end of the existingkubernetes-service-endpointslicejob'srelabel_configs(values.yaml ~line 228): Theaction: keeprule was reordered to after theserviceandnodelabel-replace rules, which deviates from the original design. While Prometheus semantics are preserved, this wastes relabeling cycles on targets that are ultimately dropped and is a regression risk for future edits. Thekeeprule should remain earlier in the pipeline.
Minor issues:
-
Missing
$end anchor in controller job keep regex (values.yaml ~line 252):.+-kubex-automation-engine-metrics-serviceshould be.+-kubex-automation-engine-metrics-service$to prevent unintended suffix matches. See inline comment. -
schemeannotation rule asymmetry (values.yaml ~line 197): Theprometheus.io/schemerelabel rule applies only to the original job; the new controller job hard-codesscheme: http. This is intentional but the comment doesn't make the asymmetry explicit, which could confuse future maintainers if the controller endpoint is ever moved to HTTPS. See inline comment.
DRY improvement opportunities:
- The YAML anchor approach introduced here (
&kubexEndpointsliceMetricsPath, etc.) is a good pattern. Consider also anchoring theprometheus.io/schemerule if a third scrape job is added in the future, to keep the pattern consistent.
Suggested next steps:
- Remove
honor_labels: truefromkubex-automation-engine-metrics-endpointslice(or explicitly justify why it is needed). - Move the
action: keependpointslice-name filter rule back to before theservice/nodelabel-replace rules inkubernetes-service-endpointslice. - Add
$to the controller job's keep regex:'.+-kubex-automation-engine-metrics-service$'. - Clarify the comment on the scheme annotation rule to document the intentional asymmetry between the two jobs.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: The PR correctly introduces YAML anchors to enable config reuse across Prometheus scrape jobs and adds a dedicated controller metrics scrape job. The approach is sound, but there are two correctness concerns in the new job: a latent port-selection dependency on an annotation, and an absence of namespace scoping that could lead to cross-namespace scraping in multi-tenant clusters.
Risk level: Medium
Critical issues:
- None
Major issues:
- Port selection depends on annotation being present (
values.yamlnew job):*kubexEndpointsliceAddressrewrites__address__only whenprometheus.io/portis annotated on the Service. If that annotation is absent or mismatched, Prometheus scrapes the wrong port silently. The job's fixed-port intent should be enforced with a hard-coded replacement rule, not an annotation-driven one. - No namespace scoping on the new job (
values.yamlnew job,keepregex): The.+-kubex-automation-engine-metrics-service$keep regex matches in all namespaces discovered bykubernetes_sd_configs. In multi-tenant clusters this can cause Prometheus to scrape identically-named Services in unintended namespaces. A namespace filter should be added tokubernetes_sd_configsor as an explicit__meta_kubernetes_namespacekeep rule.
Minor issues:
- Reordering of
action: keepin the existing job (kubernetes-service-endpointslice): Moving theendpointslice_namekeep filter after theservice/noderelabels is an unannounced behavioral change to a pre-existing job. Prometheus still works correctly, but unnecessary relabeling work is performed on targets that are subsequently dropped, and the intent is not documented. - Reuse of annotation-driven anchors in a fixed-endpoint job:
*kubexEndpointsliceMetricsPathand*kubexEndpointsliceParamLabelsintroduce annotation-driven override paths into a job explicitly designed to ignore annotation-driven scheme changes. This contradicts the documented intent and may surprise future maintainers.
DRY improvement opportunities:
- The YAML anchor strategy (
&kubexEndpointslice*/*kubexEndpointslice*) is a good pattern for this config. Consider whether future jobs (e.g., a ServiceMonitor path or additional controller releases) should also reuse these, and document the anchor conventions near the top of thescrape_configsblock.
Suggested next steps:
- Add a hard-coded
__address__replacement rule in the new job (e.g.,replacement: $1:8080) to enforce port8080independent of annotation state, and remove reliance on*kubexEndpointsliceAddressin this job. - Add a namespace keep rule before the service-name keep rule in the new job, scoping discovery to the release namespace to prevent cross-namespace scraping.
- Consider restoring the
endpointslice_namekeep filter to its original position inkubernetes-service-endpointslice, or add a comment explaining why it was moved.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall Assessment
The PR's approach — adding YAML anchors to DRY-up shared relabels, introducing a dedicated scrape job for the controller metrics service, and keeping it separated from the shared allowlist — is architecturally sound. The main risk is a port/scheme mismatch in the diff that directly contradicts the author's own validated test results and will cause up == 0 in production if shipped as-is. The remaining issues are minor maintainability concerns.
Risk level: Medium (one correctness bug, two minor maintainability issues)
Critical issues
None.
Major issues
scheme: http+ hardcoded:8443in address replacement are contradictory (values.yaml, new controller job). Port8443is a TLS port; combining it withscheme: httpwill produce TLS handshake failures on every scrape (up == 0). The author's latest comment and validation evidence confirms the correct port is8080, but the diff was not updated. This must be fixed before merge.
Minor issues
- Anchors defined but selectively unused without explanation —
&kubexEndpointsliceMetricsPath,&kubexEndpointsliceAddress, and&kubexEndpointsliceParamLabelsare defined in the shared job but not referenced in the new controller job. This is intentional (different behavior), but no comment explains which rules were deliberately omitted, creating confusion for future maintainers. scheme: httpis hardcoded with no Helm values override — operators who run the controller with TLS-enabled metrics have no way to change the scheme without modifying the chart. The sibling job handles this dynamically via annotations; the new job should at minimum expose avalues.yamlkey forschemeandport.
DRY improvement opportunities
- The YAML anchor approach adopted in this PR is a good step. The
&kubexEndpointsliceServiceLabels,&kubexEndpointsliceNamespace,&kubexEndpointsliceServiceName, and&kubexEndpointsliceNodeNameanchors are correctly reused in the new job. - Consider also anchoring the
__metrics_path__override (replacement: /metrics) if more jobs of this type are added in future.
Suggested next steps
- Fix the port: Change
replacement: '$1:8443'→replacement: '$1:8080'in the new controller job's address relabel (or parameterize via values). - Add a comment block at the top of the new job listing which shared relabels were intentionally omitted (scheme-annotation override, port-annotation override, param label map) and why — this makes future audits straightforward.
- (Optional but recommended) Add
prometheus.controllerMetrics.schemeandprometheus.controllerMetrics.porttovalues.yamlso the job is operator-configurable without a chart release.
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
Overall assessment: The PR introduces a well-structured dedicated Prometheus scrape job for the controller metrics endpoint, with good use of YAML anchors to avoid duplication across the two jobs. However, there are two correctness bugs in the new job's relabel_configs — a regex missing an end-anchor that will produce garbage __address__ values, and a keep filter running after the address-rewrite mutation rather than before it — which will cause silent scrape failures in production. There are also minor maintainability concerns around unused anchors and the potential for future double-scraping overlap between the two jobs.
Risk level: Medium (the address-regex bug and filter-ordering issue will cause the dedicated job to produce up == 0 or wrong scrape targets in practice)
Critical issues
(none)
Major issues
-
Address-rewrite regex missing end anchor (
values.yamlline ~254): The regex'(.+?)(?::\d+)?'uses a lazy quantifier with no$anchor. For an address like10.244.0.3:8080, Go's RE2 engine will satisfy the lazy match with just the first character, producing1:8080as the replacement instead of10.244.0.3:8080. Use'(.+?)(?::\d+)?$'or'([^:]+)(?::\d+)?$'. See inline comment at line 254. -
keepfilter runs after the address mutation, not before (values.yamlline ~261): In the new job'srelabel_configs, the__address__port-rewrite ($1:8080) executes on every endpointslice in the cluster before theaction: keepon__meta_kubernetes_service_namehas narrowed the target set. This means all cluster endpoints get their port overwritten, then most are dropped. The keep filter should be moved above the address-rewrite. See inline comment at line 261.
Minor issues
-
Unused YAML anchors (
values.yaml~line 200):&kubexEndpointsliceMetricsPath,&kubexEndpointsliceAddress, and&kubexEndpointsliceParamLabelsare defined but never aliased in the new job. This is either dead code or a maintenance trap (future authors completing the alias pattern would accidentally break the hard-coded port). Either remove the anchor markers or add a guard comment explaining they are intentionally not aliased. See inline comment at line 200. -
Missing guard comment on asymmetric anchor usage (
values.yaml~line 247): The new job selectively aliases*kubexEndpointsliceServiceLabels,*kubexEndpointsliceNamespace,*kubexEndpointsliceServiceName, and*kubexEndpointsliceNodeName, but explicitly inlines the path and address rules. Without a comment, future authors may "complete the pattern" by adding*kubexEndpointsliceAddress, silently overriding the port. See inline comment at line 247. -
Potential double-scrape overlap between the two jobs (
values.yaml~line 219): The sharedkubernetes-service-endpointslicejob has no explicit exclusion for the controller metrics service. If a future release changes the controller's EndpointSlice name to match the shared job's regex (which is partially wildcard-based via.*dcgm-style alternatives), both jobs will scrape the same endpoint and the shared job's allowlist will silently drop all controller metrics. Consider adding an explicitaction: dropfor.+-kubex-automation-engine-metrics-service$to the shared job. See inline comment at line 219.
DRY improvement opportunities
- The YAML anchor mechanism (
&/*) is already well-applied for the six shared relabel rules. The three anchors that are currently unused (MetricsPath,Address,ParamLabels) could be removed to keep the anchor list minimal and honest.
Suggested next steps
- Fix the address-rewrite regex to add a
$end anchor (line 254). - Move the
action: keepfilter above the__address__rewrite (line 261). - Either remove the three unused anchors (
MetricsPath,Address,ParamLabels) or add explicit "do not alias" guard comments. - Consider adding an explicit
droprule in the shared job to prevent future overlap with the controller metrics service.
| regex: '(.+?)(?::\d+)?' | ||
| replacement: '$1:8080' | ||
| # Matches any Helm release name prefix (for example 'controller-' or 'prod-'). | ||
| # This intentionally supports scraping multiple controller releases when their service names share this suffix. |
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
[Severity: Major] [Confidence: High]
Location: charts/kubex-automation-stack/values.yaml:261
Issue: The action: keep on __meta_kubernetes_service_name fires before any endpointslice-to-service name propagation from __meta_kubernetes_endpointslice_name, meaning that for multi-port services the address rewrite on line 254 ($1:8080) will execute on all discovered targets across the entire cluster before the keep filter narrows them — this is wasteful but, more critically, the address-rewrite rule (which overwrites __address__) runs at line 254 while the keep only runs at line 261. For Prometheus, all relabel_configs are evaluated in list order, so every endpoint in the cluster gets its port forced to 8080, then the keep fires. The side-effect is that if the same endpoint is also scraped by the sibling kubernetes-service-endpointslice job (because the service also matches that job's regex), it will be double-scraped; however the main concern is the ordering: move the keep filter to run before the __address__ rewrite so that the port rewrite only applies to targets that have already been selected.
Why it matters: The address-rewrite $1:8080 runs on every endpointslice target discovered cluster-wide before the keep filter narrows the set. For large clusters this inflates the set of active scrape connections (even temporarily) and can produce spurious scrape errors for unrelated services whose address was rewritten but the keep then drops. It also makes the configuration harder to reason about: readers expect the filter to come before the mutation.
Suggested fix: Re-order relabel_configs so that the service-name keep filter runs first, then the __address__ rewrite:
relabel_configs:
- target_label: __metrics_path__
replacement: /metrics
- *kubexEndpointsliceServiceLabels
- *kubexEndpointsliceNamespace
# Keep filter FIRST — before any address mutation
- source_labels: [__meta_kubernetes_service_name]
action: keep
regex: '.+-kubex-automation-engine-metrics-service$'
# Port rewrite only runs for already-selected targets
- source_labels: [__address__]
action: replace
target_label: __address__
regex: '(.+?)(?::\d+)?'
replacement: '$1:8080'
- *kubexEndpointsliceServiceName
- *kubexEndpointsliceNodeName| - target_label: __metrics_path__ | ||
| replacement: /metrics | ||
| - *kubexEndpointsliceServiceLabels | ||
| - *kubexEndpointsliceNamespace |
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
[Severity: Major] [Confidence: High]
Location: charts/kubex-automation-stack/values.yaml:254
Issue: The address-rewrite regex '(.+?)(?::\d+)?' is missing the end anchor $. Without it, (.+?) is lazy and (?::\d+)? is optional, so the overall match will succeed with (.+?) capturing just the first character of the IP, leaving the rest (including a real port) in the unmatched tail. For an input like 10.244.0.3:6443, the lazy (.+?) will match 1, satisfying the full regex, and the replacement $1:8080 produces 1:8080 rather than 10.244.0.3:8080.
Why it matters: The rewritten __address__ values will be garbage, causing every scrape by this job to fail with a connection error.
Suggested fix: Anchor the regex to the full string:
- source_labels: [__address__]
action: replace
target_label: __address__
regex: '(.+?)(?::\d+)?$'
replacement: '$1:8080'Or use a non-lazy quantifier with the anchor:
regex: '([^:]+)(?::\d+)?$'
replacement: '$1:8080'| # The target is scraped by service-name suffix so multiple controller releases can be collected when needed. | ||
| # No bearer token is required for this /metrics endpoint. | ||
| scheme: http | ||
| kubernetes_sd_configs: |
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
[Severity: Major] [Confidence: High]
Location: charts/kubex-automation-stack/values.yaml:247
Issue: The &kubexEndpointsliceMetricsPath and &kubexEndpointsliceAddress and &kubexEndpointsliceParamLabels anchors are defined in the shared job but are never referenced (*) by the new dedicated job, even though they would be harmless no-ops. However, the &kubexEndpointsliceAddress rule specifically rewrites __address__ by combining the host with the prometheus.io/port annotation. If this anchor were accidentally aliased into the new job in a future edit, it would override the explicit $1:8080 port. More importantly right now: the new job deliberately replaces __address__ with a hard-coded port but sets target_label: __metrics_path__ via an inline rule (not via the *kubexEndpointsliceMetricsPath alias). This is consistent — but should be explicitly commented to prevent a future author from "completing the pattern" by adding *kubexEndpointsliceAddress, which would break the port rewrite. Consider adding a # Note: do NOT alias *kubexEndpointsliceAddress here guard comment.
Why it matters: The asymmetric use of anchors (some aliased, some re-inlined) creates a maintenance trap. A future author completing the pattern by adding the missing aliases will silently break the port override.
Suggested fix: Add a guard comment immediately before the first alias in the new job:
relabel_configs:
# Fixed path/port — do NOT add *kubexEndpointsliceAddress or *kubexEndpointsliceMetricsPath here;
# this job intentionally hard-codes port 8080 and path /metrics.
- target_label: __metrics_path__
replacement: /metrics
- *kubexEndpointsliceServiceLabels
...| - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] | ||
| action: replace | ||
| target_label: __scheme__ | ||
| regex: (https?) |
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
[Severity: Minor] [Confidence: High]
Location: charts/kubex-automation-stack/values.yaml:197
Issue: The &kubexEndpointsliceMetricsPath, &kubexEndpointsliceAddress, and &kubexEndpointsliceParamLabels anchors are defined in the shared job but never consumed by the new job or any other section. Unused YAML anchors are dead weight and add cognitive overhead when readers scan for alias usages. If these three anchors are retained for future use, a comment should say so explicitly; otherwise they should be removed.
Why it matters: Unused anchors suggest incomplete refactoring and may mislead future readers into thinking there is a second consumer somewhere in the file.
Suggested fix: Either remove the &kubexEndpointsliceMetricsPath, &kubexEndpointsliceAddress, and &kubexEndpointsliceParamLabels anchor markers (keep the rules, just drop the &name labels), or add an inline comment:
- &kubexEndpointsliceAddress # anchor reserved for future controller HTTPS variant
source_labels: [__address__, ...]| action: labelmap | ||
| regex: __meta_kubernetes_service_label_(.+) | ||
| - source_labels: [__meta_kubernetes_namespace] | ||
| - &kubexEndpointsliceNamespace |
There was a problem hiding this comment.
CONTENT OF THIS REVIEW IS AI GENERATED
[Severity: Minor] [Confidence: High]
Location: charts/kubex-automation-stack/values.yaml:219
Issue: The action: keep regex on __meta_kubernetes_endpointslice_name in the shared kubernetes-service-endpointslice job was not updated to exclude the new controller metrics service, even though a dedicated job now exists for it. If the controller's EndpointSlice name ever happens to match the shared job's regex (e.g., a future release where the name contains kubex), the shared job and the dedicated job will both scrape it — and the shared job's metric_relabel_configs allowlist will then silently discard all controller-runtime/go/workqueue metrics.
Why it matters: Double-scraping the same endpoint wastes resources; the shared job's allowlist then drops all controller metrics that don't match the kube_*/node_*/DCGM_*/openshift_* pattern, so users would see gaps if they ever routed the controller through the shared job.
Suggested fix: Add a negative lookahead (or a separate drop rule) to the shared job's endpointslice-name keep regex so it explicitly excludes the controller metrics service name suffix:
- source_labels: [__meta_kubernetes_endpointslice_name]
action: keep
regex: '((kubex|densify)-(kube-state-metrics|prometheus-node-exporter|ephemeral-storage-collector)|.*dcgm|k8s-ephemeral-storage-metrics).*'
# Optionally, add a drop to prevent accidental overlap with the dedicated job:
- source_labels: [__meta_kubernetes_service_name]
action: drop
regex: '.+-kubex-automation-engine-metrics-service$'
|
Follow-up on the remaining review threads:
Validation rerun:
The chart now reflects the current controller scrape contract and the remaining bot concerns are either stale or intentional. |
|
Follow-up applied to the controller metrics scrape config:
This addresses the remaining review concerns around overlap and config clarity. |
1 participant
Summary
Add controller metrics exposure and scrape integration for both Prometheus Operator users and the bundled stack Prometheus path.
Make kubex-automation-stack chart able to discover the controller metrics service.
Document the new metrics scrape setup.
Testing
Prometheus query:
curl -s 'http://127.0.0.1:9090/api/v1/query?query=%7Bjob%3D%22kubernetes-service-endpointslice%22%2Cnamespace%3D%22kubex%22%2Cservice%3D%22controller-kubex-automation-engine-metrics-service%22%7D'Result:
Verified
up{service="controller-kubex-automation-engine-metrics-service"} == 1in the local Kind cluster.{ "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "up", "app_kubernetes_io_instance": "controller", "app_kubernetes_io_managed_by": "Helm", "app_kubernetes_io_name": "kubex-automation-engine", "app_kubernetes_io_version": "0.4.0", "control_plane": "controller-manager", "helm_sh_chart": "kubex-automation-engine-0.4.0", "instance": "10.244.0.16:8443", "job": "kubernetes-service-endpointslice", "namespace": "kubex", "node": "e2e-135-control-plane", "service": "controller-kubex-automation-engine-metrics-service" }, "value": [1778090924.089, "1" ] }, { "metric": { "__name__": "scrape_duration_seconds", "app_kubernetes_io_instance": "controller", "app_kubernetes_io_managed_by": "Helm", "app_kubernetes_io_name": "kubex-automation-engine", "app_kubernetes_io_version": "0.4.0", "control_plane": "controller-manager", "helm_sh_chart": "kubex-automation-engine-0.4.0", "instance": "10.244.0.16:8443", "job": "kubernetes-service-endpointslice", "namespace": "kubex", "node": "e2e-135-control-plane", "service": "controller-kubex-automation-engine-metrics-service" }, "value": [1778090924.089, "0.003503145" ] }, { "metric": { "__name__": "scrape_samples_scraped", "app_kubernetes_io_instance": "controller", "app_kubernetes_io_managed_by": "Helm", "app_kubernetes_io_name": "kubex-automation-engine", "app_kubernetes_io_version": "0.4.0", "control_plane": "controller-manager", "helm_sh_chart": "kubex-automation-engine-0.4.0", "instance": "10.244.0.16:8443", "job": "kubernetes-service-endpointslice", "namespace": "kubex", "node": "e2e-135-control-plane", "service": "controller-kubex-automation-engine-metrics-service" }, "value": [1778090924.089, "699" ] }, { "metric": { "__name__": "scrape_samples_post_metric_relabeling", "app_kubernetes_io_instance": "controller", "app_kubernetes_io_managed_by": "Helm", "app_kubernetes_io_name": "kubex-automation-engine", "app_kubernetes_io_version": "0.4.0", "control_plane": "controller-manager", "helm_sh_chart": "kubex-automation-engine-0.4.0", "instance": "10.244.0.16:8443", "job": "kubernetes-service-endpointslice", "namespace": "kubex", "node": "e2e-135-control-plane", "service": "controller-kubex-automation-engine-metrics-service" }, "value": [1778090924.089, "0" ] }, { "metric": { "__name__": "scrape_series_added", "app_kubernetes_io_instance": "controller", "app_kubernetes_io_managed_by": "Helm", "app_kubernetes_io_name": "kubex-automation-engine", "app_kubernetes_io_version": "0.4.0", "control_plane": "controller-manager", "helm_sh_chart": "kubex-automation-engine-0.4.0", "instance": "10.244.0.16:8443", "job": "kubernetes-service-endpointslice", "namespace": "kubex", "node": "e2e-135-control-plane", "service": "controller-kubex-automation-engine-metrics-service" }, "value": [1778090924.089, "0" ] } ] } }