feat(heartbeat): add api monitor type for synthetics API journeys#50802
Draft
shahzad31 wants to merge 5 commits into
Draft
feat(heartbeat): add api monitor type for synthetics API journeys#50802shahzad31 wants to merge 5 commits into
api monitor type for synthetics API journeys#50802shahzad31 wants to merge 5 commits into
Conversation
Heartbeat now recognises `monitor.type: api`, a new synthetics-driven
monitor type that runs multi-step API checks via Playwright's
`APIRequestContext` without launching Chromium.
API monitors reuse the existing synthexec runtime that browser
monitors use (same project/inline source pipeline, same SynthEvent
stream, same wrappers/summarizer plugin set, same `synthetics` data
stream). The only differences are:
* A new `x-pack/heartbeat/monitors/api` package registers the
`api` plugin and bridges it to `browser.NewSourceJob`. The
`ELASTIC_SYNTHETICS_CAPABLE` env gate (which exists to keep
browser monitors from running on machines without GUI libs) is
intentionally skipped — API journeys never launch Chromium.
* `browser/sourcejob.extraArgs` now filters browser-only CLI flags
(`--sandbox`, `--screenshots`, `--throttling`/`--no-throttling`)
when the monitor type is `api`. `--playwright-options` and
`--ignore-https-errors` are still forwarded since both apply to
`APIRequestContext`.
* `journey/network_info` events from API journeys are routed to a
new `synthetics.api.network` dataset (vs. `browser.network`),
so ingest pipelines and Kibana dashboards can branch cleanly
without inspecting `journey.type` on every doc.
* `SynthEvent.Journey.Type` (new) is propagated through `ToMap()`
so downstream consumers see `synthetics.journey.type: api`.
Older synthetics agents that don't emit the field continue to be
treated as browser for back-compat.
Central plumbing change: a new `stdfields.IsSyntheticsType(t)` helper
returns true for `browser` and `api`. The four call sites that used
to special-case `"browser"` (wrappers, summarizer, factory data-stream
auto-config, logger network-info extraction) now use the helper.
`heartbeat.config` adds a default scaling limit + `SYNTHETICS_LIMIT_API`
env var for the new type.
Requires `@elastic/synthetics` >= <TBD-version> on the host (the
release that introduces the `apiJourney(...)` DSL,
elastic/synthetics#997).
Co-authored-by: Cursor <cursoragent@cursor.com>
|
This pull request doesn't have a |
Contributor
🤖 GitHub commentsJust comment with:
|
Contributor
|
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
The api-monitor PR enabled CI's `--whole-files` golangci-lint mode on
the files it modified, surfacing seven pre-existing violations (plus
several more masked by `max-same-issues: 3`) that had been ignored
while those files were untouched on main:
* heartbeat/monitors/logger/logger.go: `extractRunInfo` was doing
five unchecked type assertions on `interface{}` values pulled out
of the event, e.g. `monType.(string)`, which trip `errcheck`'s
`check-type-assertions: true` and would panic on a malformed
event. Switched to the `, ok` form and append a typed error to
the existing aggregated-error path so callers see "monitor.type
is not a string, got <T>" instead of a runtime panic.
* heartbeat/monitors/wrappers/summarizer/summarizer.go: changed
`Summarizer.contsRemaining` from `uint16` to `int` so that
`contsRemaining += len(conts)` no longer needs the
`uint16(len(conts))` narrowing conversion that gosec G115 flags.
The field is only used as an internal "still to process" counter
compared against 0; widening to int is semantically identical.
* heartbeat/monitors/factory.go, heartbeat/config/config.go,
heartbeat/monitors/logger/logger.go,
x-pack/heartbeat/monitors/browser/sourcejob.go,
x-pack/heartbeat/monitors/browser/synthexec/synthtypes.go:
annotated the remaining `logp.L()` fallbacks with targeted
`//nolint:forbidigo` directives that explain why the call site
doesn't have a contextual `*logp.Logger` (factory predates
per-beat loggers in FactoryParams; preProcessors is invoked from
reload paths; ToMap/extraArgs/StdFields are pure mapping helpers
with no logger handle; getLogger() is the documented pre-
SetLogger fallback; DefaultConfig runs before the beat-scoped
logger is constructed). These match the dozens of other
`logp.L()` callers across `heartbeat/` that pre-date the
"accept *logp.Logger as a parameter" rule and a real refactor
is tracked separately.
* In `NewFactory`, hoisted `logger := logp.L()` to a local so the
struct literal stays `goimports`-aligned with the `//nolint`
directive sitting next to the assignment instead of inside the
field block.
Verified locally with the same filter CI uses
(`--new-from-patch ... --new=false --whole-files`) against
golangci-lint v2.5.0 on `./heartbeat/...` and
`./x-pack/heartbeat/...`: 0 issues. All affected unit tests still
pass.
Assisted-By: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
8 tasks
shahzad31
added a commit
to shahzad31/kibana
that referenced
this pull request
May 23, 2026
…ntegrations PR
Two fixes in response to review:
1. `DEFAULT_API_ADVANCED_FIELDS` no longer inherits browser values for
SCREENSHOTS / THROTTLING_CONFIG. They were semantically wrong for API
journeys (no browser → no screenshots; raw HTTP → no CDP throttling).
Defaults are now:
- SCREENSHOTS: ScreenshotOption.OFF
- THROTTLING_CONFIG: PROFILES_MAP[PROFILE_VALUES_ENUM.NO_THROTTLING]
Heartbeat's api plugin (elastic/beats#50802) also strips the matching
CLI flags, so this aligns the SO / UI / telemetry with the runtime.
Normalizer docstring and unit test updated to match.
2. Removed the speculative `monitor.type: api` branches in
`format_synthetics_policy.ts` that:
- enabled a guessed `synthetics.api.network` data stream, and
- reused the browser source.inline base64 encoding path for API.
Private-location support for API monitors requires a brand-new
`synthetics/api` input in the Fleet synthetics integration package
(elastic/integrations) — not a fork of `synthetics/browser`. Until
that integration PR lands, `inputs.find(input.type === 'synthetics/api')`
resolves to undefined, the formatter returns `hasInput: false`, and the
caller surfaces the existing "synthetics integration package needs
upgrade" warning. A TODO comment documents the follow-up and explicitly
warns against reusing browserFormatters for the eventual apiFormatters
map.
Co-authored-by: Cursor <cursoragent@cursor.com>
Draft
5 tasks
…ckage `journey/network_info` events emitted by API journeys were being routed to `synthetics.api.network`, but the Fleet `synthetics` integration package defines the companion stream as `data_stream/api_network/manifest.yml` with `dataset: api.network`. That mismatch caused Heartbeat to write to an auto-created `synthetics-synthetics.api.network-default` data stream that the Fleet-generated agent API key was not scoped to, and Elasticsearch returned `security_exception` / 403 — every API journey network event was silently dropped. Use `api.network` so the routed dataset matches the package, the agent API key includes the right `create_doc` privilege, and the documents land in `synthetics-api.network-default` like the rest of the synthetics integration. The companion-side fix — Fleet must actually enable the `api_network` stream for API monitors — lives in the linked Kibana PR (`format_synthetics_policy.ts`). Without both halves the API key is still missing the privilege. Co-authored-by: Cursor <cursoragent@cursor.com>
shahzad31
added a commit
to shahzad31/kibana
that referenced
this pull request
May 24, 2026
…et policy
The prior commit deliberately deferred Fleet wiring for `synthetics/api`
to the integrations package PR. Now that the package ships
`data_stream/api_network/manifest.yml` (`dataset: api.network`) plus a
matching `agent_input` template, the policy formatter has to actually
turn that stream on for API monitors — otherwise:
1. Fleet emits the agent API key based on the streams it sees as
`enabled: true` in the package policy. With api_network disabled
by default, the key is scoped to `synthetics-api-default` only.
2. Heartbeat (elastic/beats#50802) routes `journey/network_info`
events to `synthetics-api.network-default`, and ES rejects the
bulk write with `security_exception` / 403 — the API monitor's
network waterfall stays silently empty.
`formatSyntheticsPolicy` already toggles browser companion streams
(network, browser, screenshot) when `monitorType === BROWSER`. Add the
mirroring branch for `MonitorTypeEnum.API` so the `api.network`
stream comes online whenever an API monitor is deployed, and the
Fleet-issued API key gains the `create_doc` privilege for
`synthetics-api.network-default`.
Unit test (`enables the api.network companion stream for api monitors`)
locks in the new behaviour next to the existing browser/HTTP cases.
Co-authored-by: Cursor <cursoragent@cursor.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
monitor.type: apito Heartbeat — a synthetics-driven monitor type that runs multi-step API checks via Playwright'sAPIRequestContextwithout ever launching Chromium.Companion to
elastic/synthetics#997, which introduces theapiJourney(...)DSL on the agent side.What changed
New
apipluginx-pack/heartbeat/monitors/api/api.go— registersmonitor.type: api(aliassynthetics/api) and bridges tobrowser.NewSourceJob.ELASTIC_SYNTHETICS_CAPABLEenv gate (no Chromium needed), keeps theeuid != 0setuid guard (still spawns Node).Centralised "synthetics-driven monitor" check
stdfields.IsSyntheticsType(t)/(StdMonitorFields).IsSyntheticsType(). Returnstrueforbrowserandapi."browser"now use the helper:wrappers/wrappers.go—WrapBrowserpathwrappers/summarizer/summarizer.go— browser plugin setmonitors/factory.go— synthetics data stream auto-configmonitors/logger/logger.go— skip lightweight network-info extractionheartbeat/config/config.go— adds defaultapi: {Limit: 2}and theSYNTHETICS_LIMIT_APIenv var.Type-aware CLI invocation
browser/config.go— newTypefield on the config +(*Config).IsAPI()helper.browser/sourcejob.extraArgs— filters browser-only flags (--sandbox,--screenshots,--throttling/--no-throttling) for API journeys.--playwright-optionsand--ignore-https-errorsare still forwarded (both apply toAPIRequestContext).(*SourceJob).Plugin()is now exported so theapipackage can reuse the pipeline.Event routing
synthexec/synthtypes.go—Journey.Typeis a new optional field;(Journey).IsAPI()helper.ToMap()omitstypewhen empty so older agent docs aren't reshaped.synthexec/enrich.go—journey/network_infoevents from API journeys land insynthetics.api.network(new dataset) instead ofbrowser.network. Legacy/unknown journeys keep going tobrowser.networkfor back-compat.Tests (all green)
stdfields/stdfields_test.go— table-drivenTestIsSyntheticsType.browser/sourcejob_test.go—TestExtraArgsForAPIMonitor(filters apply) +TestExtraArgsForBrowserMonitorUnchanged(regression guard).synthexec/enrich_test.go—TestEnrichAPIJourneyDatasetRouting(api →synthetics.api.network) +TestEnrichLegacyJourneyDefaultsToBrowser(nojourney.type→browser.network).synthexec/synthtypes_test.go—TestJourneyTypePropagationcoversToMap,IsAPI, JSON unmarshal.Compatibility
@elastic/synthetics >= <TBD>(the release that shipsapiJourney(...)). Older agents silently won't emitjourney.type, andjourney/network_infofor any of their journeys keeps going tobrowser.network— same behavior as before, no regression.synthetics.api.network. Until ILM / dashboards in Kibana ship support, those docs land but won't be surfaced anywhere bespoke.browsermonitors: byte-compatible. The only branching change isType != "browser"→!IsSyntheticsType(), which is logically identical when the only type in the set isbrowser.TestExtraArgsForBrowserMonitorUnchangedpins that the browser CLI invocation is unchanged.Out of scope (follow-ups)
docs/reference/synthetics-monitor.asciidoc) — covered separately once the synthetics-side docs land.dev-tools/packaging/templates/docker/Dockerfile.tmpl— gated on@elastic/syntheticsreleasing.monitor.type: apiin the monitor list / waterfall / state.Test plan
go build -tags synthetics ./heartbeat/... ./x-pack/heartbeat/...go test -tags syntheticson all touched packages.apiJourneyagainsthttpbin.org, verify events land insynthetics.api.network.Linked issues / PRs
apiJourneyDSL + capture pipeline.