Skip to content

feat(heartbeat): add api monitor type for synthetics API journeys#50802

Draft
shahzad31 wants to merge 5 commits into
elastic:mainfrom
shahzad31:synth/api-monitor-type
Draft

feat(heartbeat): add api monitor type for synthetics API journeys#50802
shahzad31 wants to merge 5 commits into
elastic:mainfrom
shahzad31:synth/api-monitor-type

Conversation

@shahzad31

Copy link
Copy Markdown
Contributor

Summary

Adds a new monitor.type: api to Heartbeat — a synthetics-driven monitor type that runs multi-step API checks via Playwright's APIRequestContext without ever launching Chromium.

Companion to elastic/synthetics#997, which introduces the apiJourney(...) DSL on the agent side.

# heartbeat.yml
heartbeat.monitors:
  - type: api
    id: orders-api-health
    name: Orders API health
    schedule: '@every 1m'
    source:
      local:
        path: '/etc/synthetics/api-checks'

What changed

New api plugin

  • x-pack/heartbeat/monitors/api/api.go — registers monitor.type: api (alias synthetics/api) and bridges to browser.NewSourceJob.
  • Skips the ELASTIC_SYNTHETICS_CAPABLE env gate (no Chromium needed), keeps the euid != 0 setuid guard (still spawns Node).

Centralised "synthetics-driven monitor" check

  • New stdfields.IsSyntheticsType(t) / (StdMonitorFields).IsSyntheticsType(). Returns true for browser and api.
  • Four call sites that previously special-cased "browser" now use the helper:
    • wrappers/wrappers.goWrapBrowser path
    • wrappers/summarizer/summarizer.go — browser plugin set
    • monitors/factory.go — synthetics data stream auto-config
    • monitors/logger/logger.go — skip lightweight network-info extraction
  • heartbeat/config/config.go — adds default api: {Limit: 2} and the SYNTHETICS_LIMIT_API env var.

Type-aware CLI invocation

  • browser/config.go — new Type field on the config + (*Config).IsAPI() helper.
  • browser/sourcejob.extraArgs — filters browser-only flags (--sandbox, --screenshots, --throttling/--no-throttling) for API journeys. --playwright-options and --ignore-https-errors are still forwarded (both apply to APIRequestContext).
  • (*SourceJob).Plugin() is now exported so the api package can reuse the pipeline.

Event routing

  • synthexec/synthtypes.goJourney.Type is a new optional field; (Journey).IsAPI() helper. ToMap() omits type when empty so older agent docs aren't reshaped.
  • synthexec/enrich.gojourney/network_info events from API journeys land in synthetics.api.network (new dataset) instead of browser.network. Legacy/unknown journeys keep going to browser.network for back-compat.

Tests (all green)

  • stdfields/stdfields_test.go — table-driven TestIsSyntheticsType.
  • browser/sourcejob_test.goTestExtraArgsForAPIMonitor (filters apply) + TestExtraArgsForBrowserMonitorUnchanged (regression guard).
  • synthexec/enrich_test.goTestEnrichAPIJourneyDatasetRouting (api → synthetics.api.network) + TestEnrichLegacyJourneyDefaultsToBrowser (no journey.typebrowser.network).
  • synthexec/synthtypes_test.goTestJourneyTypePropagation covers ToMap, IsAPI, JSON unmarshal.
ok  github.com/elastic/beats/v7/heartbeat/monitors/stdfields
ok  github.com/elastic/beats/v7/heartbeat/monitors/wrappers
ok  github.com/elastic/beats/v7/heartbeat/monitors/wrappers/summarizer
ok  github.com/elastic/beats/v7/heartbeat/monitors/logger
ok  github.com/elastic/beats/v7/heartbeat/config
ok  github.com/elastic/beats/v7/x-pack/heartbeat/monitors/browser
ok  github.com/elastic/beats/v7/x-pack/heartbeat/monitors/browser/synthexec

Compatibility

  • Synthetics agent: requires @elastic/synthetics >= <TBD> (the release that ships apiJourney(...)). Older agents silently won't emit journey.type, and journey/network_info for any of their journeys keeps going to browser.network — same behavior as before, no regression.
  • Stack / ingest pipelines: introduces a new dataset synthetics.api.network. Until ILM / dashboards in Kibana ship support, those docs land but won't be surfaced anywhere bespoke.
  • Existing browser monitors: byte-compatible. The only branching change is Type != "browser"!IsSyntheticsType(), which is logically identical when the only type in the set is browser. TestExtraArgsForBrowserMonitorUnchanged pins that the browser CLI invocation is unchanged.

Out of scope (follow-ups)

  • Heartbeat docs (docs/reference/synthetics-monitor.asciidoc) — covered separately once the synthetics-side docs land.
  • Bundled synthetics version bump in dev-tools/packaging/templates/docker/Dockerfile.tmpl — gated on @elastic/synthetics releasing.
  • An end-to-end integration test against a real HTTPS server — depends on the synthetics agent being installed in CI.
  • Kibana-side handling of monitor.type: api in the monitor list / waterfall / state.

Test plan

  • go build -tags synthetics ./heartbeat/... ./x-pack/heartbeat/...
  • go test -tags synthetics on all touched packages.
  • Local end-to-end smoke: heartbeat container with a real apiJourney against httpbin.org, verify events land in synthetics.api.network.
  • CI on this branch.

Linked issues / PRs

Heartbeat now recognises `monitor.type: api`, a new synthetics-driven
monitor type that runs multi-step API checks via Playwright's
`APIRequestContext` without launching Chromium.

API monitors reuse the existing synthexec runtime that browser
monitors use (same project/inline source pipeline, same SynthEvent
stream, same wrappers/summarizer plugin set, same `synthetics` data
stream). The only differences are:

  * A new `x-pack/heartbeat/monitors/api` package registers the
    `api` plugin and bridges it to `browser.NewSourceJob`. The
    `ELASTIC_SYNTHETICS_CAPABLE` env gate (which exists to keep
    browser monitors from running on machines without GUI libs) is
    intentionally skipped — API journeys never launch Chromium.

  * `browser/sourcejob.extraArgs` now filters browser-only CLI flags
    (`--sandbox`, `--screenshots`, `--throttling`/`--no-throttling`)
    when the monitor type is `api`. `--playwright-options` and
    `--ignore-https-errors` are still forwarded since both apply to
    `APIRequestContext`.

  * `journey/network_info` events from API journeys are routed to a
    new `synthetics.api.network` dataset (vs. `browser.network`),
    so ingest pipelines and Kibana dashboards can branch cleanly
    without inspecting `journey.type` on every doc.

  * `SynthEvent.Journey.Type` (new) is propagated through `ToMap()`
    so downstream consumers see `synthetics.journey.type: api`.
    Older synthetics agents that don't emit the field continue to be
    treated as browser for back-compat.

Central plumbing change: a new `stdfields.IsSyntheticsType(t)` helper
returns true for `browser` and `api`. The four call sites that used
to special-case `"browser"` (wrappers, summarizer, factory data-stream
auto-config, logger network-info extraction) now use the helper.
`heartbeat.config` adds a default scaling limit + `SYNTHETICS_LIMIT_API`
env var for the new type.

Requires `@elastic/synthetics` >= <TBD-version> on the host (the
release that introduces the `apiJourney(...)` DSL,
elastic/synthetics#997).

Co-authored-by: Cursor <cursoragent@cursor.com>
@botelastic botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 19, 2026
@botelastic

botelastic Bot commented May 19, 2026

Copy link
Copy Markdown

This pull request doesn't have a Team:<team> label.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
  • /test : Run the Buildkite pipeline.

@mergify

mergify Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @shahzad31? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

The api-monitor PR enabled CI's `--whole-files` golangci-lint mode on
the files it modified, surfacing seven pre-existing violations (plus
several more masked by `max-same-issues: 3`) that had been ignored
while those files were untouched on main:

  * heartbeat/monitors/logger/logger.go: `extractRunInfo` was doing
    five unchecked type assertions on `interface{}` values pulled out
    of the event, e.g. `monType.(string)`, which trip `errcheck`'s
    `check-type-assertions: true` and would panic on a malformed
    event. Switched to the `, ok` form and append a typed error to
    the existing aggregated-error path so callers see "monitor.type
    is not a string, got <T>" instead of a runtime panic.

  * heartbeat/monitors/wrappers/summarizer/summarizer.go: changed
    `Summarizer.contsRemaining` from `uint16` to `int` so that
    `contsRemaining += len(conts)` no longer needs the
    `uint16(len(conts))` narrowing conversion that gosec G115 flags.
    The field is only used as an internal "still to process" counter
    compared against 0; widening to int is semantically identical.

  * heartbeat/monitors/factory.go, heartbeat/config/config.go,
    heartbeat/monitors/logger/logger.go,
    x-pack/heartbeat/monitors/browser/sourcejob.go,
    x-pack/heartbeat/monitors/browser/synthexec/synthtypes.go:
    annotated the remaining `logp.L()` fallbacks with targeted
    `//nolint:forbidigo` directives that explain why the call site
    doesn't have a contextual `*logp.Logger` (factory predates
    per-beat loggers in FactoryParams; preProcessors is invoked from
    reload paths; ToMap/extraArgs/StdFields are pure mapping helpers
    with no logger handle; getLogger() is the documented pre-
    SetLogger fallback; DefaultConfig runs before the beat-scoped
    logger is constructed). These match the dozens of other
    `logp.L()` callers across `heartbeat/` that pre-date the
    "accept *logp.Logger as a parameter" rule and a real refactor
    is tracked separately.

  * In `NewFactory`, hoisted `logger := logp.L()` to a local so the
    struct literal stays `goimports`-aligned with the `//nolint`
    directive sitting next to the assignment instead of inside the
    field block.

Verified locally with the same filter CI uses
(`--new-from-patch ... --new=false --whole-files`) against
golangci-lint v2.5.0 on `./heartbeat/...` and
`./x-pack/heartbeat/...`: 0 issues. All affected unit tests still
pass.

Assisted-By: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
shahzad31 added a commit to shahzad31/kibana that referenced this pull request May 23, 2026
…ntegrations PR

Two fixes in response to review:

1. `DEFAULT_API_ADVANCED_FIELDS` no longer inherits browser values for
   SCREENSHOTS / THROTTLING_CONFIG. They were semantically wrong for API
   journeys (no browser → no screenshots; raw HTTP → no CDP throttling).
   Defaults are now:
     - SCREENSHOTS: ScreenshotOption.OFF
     - THROTTLING_CONFIG: PROFILES_MAP[PROFILE_VALUES_ENUM.NO_THROTTLING]
   Heartbeat's api plugin (elastic/beats#50802) also strips the matching
   CLI flags, so this aligns the SO / UI / telemetry with the runtime.
   Normalizer docstring and unit test updated to match.

2. Removed the speculative `monitor.type: api` branches in
   `format_synthetics_policy.ts` that:
     - enabled a guessed `synthetics.api.network` data stream, and
     - reused the browser source.inline base64 encoding path for API.
   Private-location support for API monitors requires a brand-new
   `synthetics/api` input in the Fleet synthetics integration package
   (elastic/integrations) — not a fork of `synthetics/browser`. Until
   that integration PR lands, `inputs.find(input.type === 'synthetics/api')`
   resolves to undefined, the formatter returns `hasInput: false`, and the
   caller surfaces the existing "synthetics integration package needs
   upgrade" warning. A TODO comment documents the follow-up and explicitly
   warns against reusing browserFormatters for the eventual apiFormatters
   map.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ckage

`journey/network_info` events emitted by API journeys were being
routed to `synthetics.api.network`, but the Fleet `synthetics`
integration package defines the companion stream as
`data_stream/api_network/manifest.yml` with `dataset: api.network`.
That mismatch caused Heartbeat to write to an auto-created
`synthetics-synthetics.api.network-default` data stream that the
Fleet-generated agent API key was not scoped to, and Elasticsearch
returned `security_exception` / 403 — every API journey network
event was silently dropped.

Use `api.network` so the routed dataset matches the package, the
agent API key includes the right `create_doc` privilege, and the
documents land in `synthetics-api.network-default` like the rest of
the synthetics integration.

The companion-side fix — Fleet must actually enable the
`api_network` stream for API monitors — lives in the linked Kibana
PR (`format_synthetics_policy.ts`). Without both halves the API key
is still missing the privilege.

Co-authored-by: Cursor <cursoragent@cursor.com>
shahzad31 added a commit to shahzad31/kibana that referenced this pull request May 24, 2026
…et policy

The prior commit deliberately deferred Fleet wiring for `synthetics/api`
to the integrations package PR. Now that the package ships
`data_stream/api_network/manifest.yml` (`dataset: api.network`) plus a
matching `agent_input` template, the policy formatter has to actually
turn that stream on for API monitors — otherwise:

  1. Fleet emits the agent API key based on the streams it sees as
     `enabled: true` in the package policy. With api_network disabled
     by default, the key is scoped to `synthetics-api-default` only.
  2. Heartbeat (elastic/beats#50802) routes `journey/network_info`
     events to `synthetics-api.network-default`, and ES rejects the
     bulk write with `security_exception` / 403 — the API monitor's
     network waterfall stays silently empty.

`formatSyntheticsPolicy` already toggles browser companion streams
(network, browser, screenshot) when `monitorType === BROWSER`. Add the
mirroring branch for `MonitorTypeEnum.API` so the `api.network`
stream comes online whenever an API monitor is deployed, and the
Fleet-issued API key gains the `create_doc` privilege for
`synthetics-api.network-default`.

Unit test (`enables the api.network companion stream for api monitors`)
locks in the new behaviour next to the existing browser/HTTP cases.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs_team Indicates that the issue/PR needs a Team:* label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant