From 6c5e3f11a8c65d5ed9a9e9a62d20137774b879f5 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 29 Jan 2026 14:46:58 +0800 Subject: [PATCH 01/22] Proposal for label/values discovery API Signed-off-by: Andrew Hall --- proposals/0000-new-labels-values-api.md | 485 ++++++++++++++++++++++++ 1 file changed, 485 insertions(+) create mode 100644 proposals/0000-new-labels-values-api.md diff --git a/proposals/0000-new-labels-values-api.md b/proposals/0000-new-labels-values-api.md new file mode 100644 index 0000000..04c8c97 --- /dev/null +++ b/proposals/0000-new-labels-values-api.md @@ -0,0 +1,485 @@ +## V2 API for labels and values discovery + +* **Owners:** + * Ismail Simsek [@itsmylife](https://github.com/itsmylife) [ismail.simsek@grafana.com](mailto:ismail.simsek@grafana.com) + * Andrew Hall [@tcp13equals2](https://github.com/tcp13equals2) [andrew.hall@grafana.com](mailto:andrew.hall@grafana.com) + +* **Implementation Status:** Not implemented + +* **Related Issues and PRs:** + +* **Other docs or links:** +* [Grafana - Performance improvements for high cardinality metrics #100376](https://github.com/grafana/grafana/issues/100376) +* [Grafana - Autocomplete desired API flow](https://mermaid.live/edit#pako:eNrFVNtu00AQ_ZXVPlQgXN9ix4lFgrgJ-lBU1PICQdbGO0lW2LtmL2ncKP_OrlMLo5aiSpXwg6Udj8-ZOWd29rgUFHCOFfw0wEt4x8haknrBkX0aIjUrWUO4Rl8UyLvR10aLUtRNBRru-XpxhohCnw3IFr0h5Q_g1AWPmce3wz2dz4dAObpqG1BogcvGLPAxb5jg8i_OcvTh_RUKSMOCbRxUZAmVCihTpdiCfPVyKYN5I2HFdjMLc1KxmulZHJ4oIXWxbGcreWy5PWG8rAyFQgtNqpmW5rYVy3F6p7T9llQGVI6-deUVRpE1LLCHbo-aVeyGaCa4C_q-_93rSunAcxSloYc2RBW1kBbNkR2GcvzRpiV38uTociOukRYNikMkQZlKK_TCMlo01AF7SLkUB9oLdq-wH5l2unZ4510yYhzpDSAyyENUioaKa_5fpH8aUrFaKehY_07_CJsbkCVw_dvo0qqnHmnxA-Zy2Omhu8-SMHDuVoJQoM-HI_JJWIdcz53FXqdMr9q_blPxVPepMA_qij28lozi3HXv4RpkTdwR7x37AtuJq-3w2cIxhRWxLbu6DvY3uzi-ClH3f0ph1pv-YBpKdL-i-qAb3MuWl_3ZAuB8j3c4j7Kpn4zjcBJFozjM4lHm4RbncRj56SSaJOk4TUfhOMkOHr7pOEM_m0zjcJSMJ3EWjqKph4EyLeT5cUt2y9IWZfcYyLfCcI3zNE0OvwCMfMHW) +* [Grafana - Metrics Browser/Explore desired API flow](https://mermaid.live/edit#pako:eNqllm1v2zYQx7_KgS-KBlVs-UGILMwZ1u4BxZatW7Y3mweDlk4WUUlUSSqJYvi790hKgfNQN23zwokux__d_e581I6lMkOWMI0fWqxT_FHwreLVqgb6abgyIhUNrw38o1E9tr5uRZmhAq7hAo0SqR5Mj31_ePfW-v3ZourgNU_fY51Zo_f0nzbK6fl5r5HAHw3WGqpeeqPk9V0avQ95k0YCv_z0N4x5I8ZX03HvP86ETuUVqu-_26jxeSkqYZZR-ELUadlmuCY3nnHDl0a16EV_lwZBiW1hQObghP9C06qDJK6FKcB0DerAyba1MDoATsVsuBYpiDqXXo3Onx5Ws-s1EvhvV_MKE1ixwphmrSx9bfTaSMPLFfPKNoh1SWVbGyqbBWCDWdNwwNpGo9Heff7fH7MaCUymYRhAwfW6kop0bJH7--goNws8gctCXkMulDYQhXeVvrQS4OROjjXpEktMzUDoUzV9tmsl32D5sGlec_mE4ouhn59g7eUI9YqRSCEzy2rFtOGm1Ws79t5AU9hIUZse5X2I8SHBnJf6OMIY-BUXFLlE8PGfw815PkztebjGB2fGV7wkPt_Kzas4btR-z2gezv0fkbc8Hrboy0BNor5qH-2QkvsK2gFwvAJX93GKXKUFarB9rlYMXubt7W0H2tlPvnJZOI0lCT7A9cSC-NmFq7hxWdARqbrA_l63mm_Rc0plbbioUa0HBzTp6LNrwtVE7l7K98Aq343ZsCvslrgfYb3pDOonuzX_wm7NI3A8Tn2RGSjUbWmODvebUqTvbVN-kzyDC4pErZE1bUdhBC-HJfOV7ZF5rtEt8-ds9SOAiY3deJTety_QGm8e7M8wHLsdWhIDzE6eHvNezE_6r9gl8LZnZI_5-ybDnBNxcNXCKzgccXjlsm74VtTcCGK8pGmsaV7cHrIsiFV_ceUl3ghn7nl2LGBbJTKW2PoCmi5VcfvIdjbRFTMFVtQ7e-v0WdjltKdjdKv_K2U1nFSy3RbDQ9tQ3OFtYjDy1sjLrk6HZxJgyY7dsGQymY2m03C6mJydTeNodhYHrGMJPY_O4sUink_mYRjF0T5gty7khNo2DSezeE7_mi3ixTxgmAkj1YV_oXHvNZQUrXdUb-wNypJZvP8IvLvw7A) +* [Grafana - Metrics Drilldown desired API flow](https://mermaid.live/edit#pako:eNq1Vl2T2jYU_St39LCTTFgwLF7AU7aTdttMp036se1LQ4YRlsCaGsuRZIjL8N9zJdngEJNl28k-7GBb9-uce67ujsSScRIRzd8XPIv5vaArRdezDPAvp8qIWOQ0M_CX5urzt_dKpCmT2wyohtfcKBHr48vPz7_87Sd78veCqxK-o_E_PGP2pT_p_9tI13d3By8R_JrzTAM7xGJUJwtJFfMGb6ThoMQqMSCX0LD7RVIGRuZAYyM2HNY-QW91OIexMIMIXv3wJ_RoLnqbQa862WNCx3LD1bffLFTvLhVrYab9ILjSUpn5opw6x8KUVyKL04LxORpSRg2dGlVwHwidX39az65yH8HbGUmMyefKwq-NnhtpaDojHRdvRtZ8LVU5LzRdcYxnuMZv0O123_kT7ngE_UEQdCCheo7HeQQ2-P60TMzBQhvBQyK3gFXAWmrTis05Ih54ymOjq8NwJvkL8E3pgqen8Hqv0xafVx758Ah8jOyLjKaIfVsTuIB_cFMobBwfDKwpZ7AoXbCGA3jmgNjQFEPCUihtnp-lzjtzzGHv5lJkxlIyI9pQU-i51VOTP5NI1srZuEnYkqb6McbGdSFLqbwYRLaCo84eI81ZnyZ6KVe9hlXPI3UhaU25LJUfM-VZeL1rBy_2tEd2GAz9j7B-EwY37ToInwpqPwRfGdjKqh5o4un6yvaoQ7bj0Pky3i_zPBWukVKDZg3gplVRj4wt6xK2OC41Di_PtKMZloUyCVeu4q0wCfAPQhvbBVUs6kKzryBA_52aOHn7ru3ArqXM_VVjaobB0yV1VM-MiAwj4AXVyvvoHO2PTwW6oQJ_ptzKyvlE9C19rliLLUJe4fvlVhoBZUwYIXGq1N6Rv4qaRXmZSk99QBOUi9Vam1wq1f9N739Wucu7QDi059k950qyAtFoJXv4ZJEPQ6gB0fDME-JugjZxPm_ylErcHn7EectpnPjp4D-c8tCo7hXPuKLYd7jxIH92WreE6dQZTT_F4Gu5P0LqI-D3ZqEt0N3XWxa4WePC-cvHrlQHQNuH5cGfn5g_8xK5wP6wgkLc672pd-gVeFEp5ZpuqeJV89fzyd_ZLzCJFV7aVh8wxVRiBI0d10HtM3W3uUI9bezKWS02pENWSjAS2d2og6NFral9JDub-oygytc4XHC4E8aXtEid3PZohoX_LeW6tlSyWCX1Q5HjslcvzfVLWhj5UGZx_YwOSLQjH0h0G3aHwWQcjkfBzW3_ZjDukJJE16Og3x0Owttw0h9NbsejcN8h_7qQ_e7NKBj1g-Fk0L9B0-GgQzgOCKle-73dre-YFNLB1feyyAyJJvuPHWAAkw) +* [Grafana - Possible streaming API flow](https://mermaid.live/edit#pako:eNqNVU1v2zgQ_SsDXisnlpTEqQ4BEqsttkDT7qq5LHyZSGObXYlUScpJGuS_71CU6tj5qg-mSLwZvnnzwXtR6opEJiz97EiVlEtcGWwWCvjXonGylC0qB_NaEi9o4ZPBJSocTp4i88KjvhndkFtTZyFHh4XuTElPwR62B7_A8j9S1VPs9yK_8Nh-LZw2uGKPAXepHYHekBloRR6UebettlhDChP4oKqJ0xNe2NoQNlKt4Ea6NVzmn4uvl6Ov4GFydpYXGXz68B0OsZWHm-Swxmuq7WElbemvCmgsndwg354X4SD85wU78FFl8FGbGzQV2N-XGq-2dXsOPPqxC79nJyGSvzsydxAYwEYiSEcGWYQ9Jx79high2kEDrwuWa6ilIpAWpKqoZfl9soMoW0LefPI7rAt0bBc_4jpIFkhm8ENfR-zPOuTCikBhQ7bFsQpYIMYHYn9q0wdk5GrtQC9htP2GxlIfQMTKMnUDsmmokqxHffdWAMnLAbS6iqDUyiH7NsyGe-U18i_jnyd-3nqlwWmmbbva2Qi6tvJZvPrrLdrpy7QtmY306rHzVkvOO_je0tVr3F8xGiqsdnDeOV3qpq2JOU5g7pNUAy65FMF2yyV3qq-bIZpgt9NR461X1ltQTaWjCjCweIK_9VGdX2vj5iyq0XVN5gD9fgvlaG6DLvNacxmw_Iq9Sq22mF6k21D8PDfaoXd2MNteCbh_yPYjy8LSEFU-mXCD1tP1CnRuzz58sZQEX8gZWfIg62Tti3EythonelBvx3Q3vR-l4nl17ZO8F8Femg2t2E0EvzR37juo_LKscbWjzTOZftvu-WodghjzH3FF1BDH40wyVJLc0FBk_QAPnxXtDae9w-3Ye3SYFyISKyMrkTnTUSQaMg36rbj34IXgx6Khhcj4s6IlcsEtxEI9sBk_Fv9q3YyWRner9bgJ_TW8cuMhclkXd6oc9-xAZPfiVmSTeDo9SNLZ6TSezpJZcnRyHIk7Pk_ShM-PT2azeJam8TQ-eojEr_7W5GB6Eqez4_fp-9P49Oj0eBYJHkY8qr-Et7Z_cplXP6rmulOOjZL04X-xgmzc) + +> TL;DR: This design doc proposes a richer API for searching and discovering metric names, labels, and values to power web applications. + +## Why + +The existing Prometheus [metadata API endpoints](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-metadata) are used by web applications (including Grafana) to query series metadata, such as label names and label values. + +These endpoints are heavily relied upon by the Grafana web application for core discovery workflows, including: + +* __Autocomplete__ — triggered as users type queries +* __Metrics Explorer / Builder__ — a click-based interface for browsing metrics and label dimensions +* __Metrics Drilldown__ — dynamically generating metric panels by iterating over label values + +As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, these existing endpoints have increasingly become a limiting factor. + +In practice, this manifests as performance bottlenecks in the Grafana UI and constrains the implementation of user-requested discovery features, particularly those that require iterative or interactive exploration of large metadata sets. + +Grafana has attempted to mitigate these limitations through a range of client-side optimizations and workarounds (see [#100376](https://github.com/grafana/grafana/issues/100376) +). However, these approaches add complexity and cannot fully address the underlying scalability and efficiency constraints of the current metadata APIs. + +### Pitfalls of the current solution + +Pitfalls of the current metadata endpoints; + +* __Limited client-side filtering__: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. +* __Limited matching capabilities__: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). +* __Unordered `limit` results__: `limit` returns an arbitrary subset of values rather than returning the most relevant or most frequent values, which causes drilldown UIs to display incomplete or unrepresentative results. +* __No metadata about further results__: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). +* __No pagination/cursors__: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. +* __Fragmented calls for full context__: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. +* __No streaming / incremental delivery__: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and perceived latency. + +## Goals + +Effective data queries cannot be written if what needs to be queried cannot be identified first. Improving metric name, label name and label value discovery reduces Mean Time to Execute Query (MTEQ). This can be rephrased as “users spend less time hunting for the right metrics and more time analyzing data.” + +* __Faster metric name and label identification__ - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" +* __Better autocomplete performance__ - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side +* __Scalable high-cardinality handling__ -server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion +* __Reduced bandwidth/resource consumption__ - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user +* __Improved AI/ML agent efficiency__ - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times + +### Audience + +* Operators seeking to discover metrics, labels and values + +## Non-Goals + +* Changing/enhancing/deprecating any existing Prometheus API endpoints related to labels, values and metadata +* Addressing OpenTelemetry hidden label limitations. Resource attributes stored in `target_info` rather than on every series and therefor not surfaced in the existing labels' endpoint. + +## How + +The following API endpoints are proposed. Each endpoint should return data in a NDJSON / chunked encoding streaming format. + +### `GET /api/v1/search/metrics` + +An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric. + +#### Request + +**Method:** `GET` `POST` +**Path:** `/api/v1/search/metrics` + +**Query parameters:** + +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------------|----------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `search` | string | No | N/A | The search string to be used for matching against metric names.

Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold.

The matching score of a fuzzy match must be >= this value.

A value of 100 disables any fuzz matching. | +| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | +| `sort_by` | alpha
cardinality
frequency
score | No | N/A | A flag to define how matching metrics should be sorted in the response.

* __alpha__ - metrics are sorted by alphabetical order
* __cardinality__ - metrics are sorted by their cardinality
* __frequency__ - metrics are sorted by their frequency of use
* __score__ - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. +| `sort_dir` | asc
dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. +| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. +| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. +| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. +| `start` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `end` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | +| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk.
A value of 0 indicates that the server can determine the batch size, which may be variable. | +| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | + | + +**Notes:** +- If `search` is omitted or empty, all metric names are matched. +- The fuzzy search could be a Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to a 0..100 threshold. A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100` +- Support for `cursor` based pagination is desirable but not essential. +- `start` and `end` could have default values set which align to a reasonable look-back period. ie last 24 hours +- `include_*` should use `json:"omitempty"` to not serialize the enriched attributes if they have not been requested / initialised + +Example fuzzy matching; + +* Jaro-Winkler, mimir vs mimer = 0.953 +* Levenshtein, mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8 + +#### Response + +**Status codes:** + +Use existing Prometheus API status codes. + +**Body (JSON):** + +* Content-Type: application/x-ndjson; charset=utf-8 + +##### Example of NDJSON batched result set - no include_* flags set + +```ndjson +{ + "results": [ + { "name": "go_gc_duration_seconds" }, + { "name": "go_gc_duration_seconds_count" }, + { "name": "go_gc_duration_seconds_sum" } + ] +} + +{ + "results": [ + { "name": "go_gc_gogc_percent" }, + { "name": "go_gc_gomemlimit_bytes" } + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +##### Example of NDJSON batched result set - with include_* flags set + +```ndjson +{ + "results": [ + { + "name": "activity_tracker_failed_total", + "frequency": 1003, + "cardinality": 10, + "type": "counter", + "help": "How many times has activity tracker failed to insert new activity", + "unit": "" + }, + { + "name": "activity_tracker_free_slots", + "frequency": 4, + "cardinality": 50, + "type": "gauge", + "help": "Number of free slots in activity file.", + "unit": "" + } + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +##### Example of pagination token (if supported) + +If cursor based pagination is supported, the `has_more` attribute is not required. The presence of the `cursor` attribute in the response indicates that there is more results available. + +```ndjson +{ + "results": [ + ... + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "cursor": "" +} +``` + +##### Example no results matching + +```ndjson +{ + "results": [ ], +} + +{ + "status": "success", + "has_more": false +} +``` + +##### Example API error + +As per existing endpoints. + +```ndjson +{ + "status": "error", + "errorType": "bad_data", + "error": "1:4: parse error: unexpected \"(\"", +} +``` + +### `GET /api/v1/search/labels` + +An endpoint specific to searching for label names. + +#### Request + +**Method:** `GET` `POST` +**Path:** `/api/v1/search/labels` + +**Query parameters:** + +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | +| `search` | string | No | N/A | The search string to be used for matching against label names.

Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha
cardinality
frequency
score | No | N/A | A flag to define how matching labels should be sorted in the response.

* __alpha__ - labels are sorted by alphabetical order
* __cardinality__ - labels are sorted by their value cardinality
* __frequency__ - labels are sorted by their frequency of use
* __score__ - labels are sorted by a matching score.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. +| `sort_dir` | asc
dsc | No | asc | As per above endpoint +| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. +| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. +| `start` | rfc3339
unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339
unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | +| + +**Notes:** +- If `search` is omitted or empty, all label names are matched. + +#### Response + +**Status codes:** + +Use existing Prometheus API status codes. + +**Body (JSON):** + +* Content-Type: application/x-ndjson; charset=utf-8 + +##### Example of NDJSON batched result set - no include_* flags set + +```ndjson +{ + "results": [ + { "name": "cluster" }, + { "name": "container" }, + { "name": "instance" } + ] +} + +{ + "results": [ + { "name": "job" }, + { "name": "namespace" } + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +##### Example of NDJSON batched result set - with include_* flags set + +```ndjson +{ + "results": [ + { + "name": "cluster", + "frequency": 1003, + "cardinality": 10 + }, + { + "name": "container", + "frequency": 4, + "cardinality": 50 + } + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +### `GET /api/v1/search/values` + +An endpoint specific to searching for label values. + +#### Request + +**Method:** `GET` `POST` +**Path:** `/api/v1/search/values` + +**Query parameters:** + +| Name | Type | Required | Default | Description | +|-----------------------|------------------------------------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | +| `label` | string | Yes | N/A | The label the user is requesting values for. | +| `search` | string | No | N/A | The search string to be used for matching against label values.

Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha
frequency
score | No | N/A | A flag to define how matching values should be sorted in the response.

* __alpha__ - values are sorted by alphabetical order
* __frequency__ - values are sorted by their frequency of use
* __score__ - values are sorted by a matching score.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. +| `sort_dir` | asc
dsc | No | asc | As per above endpoint +| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. +| `start` | rfc3339
unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339
unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | +| + +**Notes:** +- If `search` is omitted or empty, all label values are matched. +- The `label` parameter has been deliberately added as a required query parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. + +#### Response + +**Status codes:** + +Use existing Prometheus API status codes. + +**Body (JSON):** + +* Content-Type: application/x-ndjson; charset=utf-8 + +### Example of NDJSON batched result set - no include_* flags set + + +```ndjson +{ + "results": [ + { "name": "cluster1" }, + { "name": "cluster2" }, + { "name": "cluster3" } + ] +} + +{ + "results": [ + { "name": "cluster4" }, + { "name": "cluster5" } + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +### Example of NDJSON batched result set - with include_* flags set + +```ndjson +{ + "results": [ + { + "name": "cluster1", + "frequency": 1003 + }, + { + "name": "cluster1", + "frequency": 4 + } + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +### Additional notes + +For the `frequency` and `cardinality` enrichments, an idea is to support a "maximum reported threshold". + +For instance, the API request could include a `max-reported-frequency=100`. As the server side indexes the frequency of use it can stop once it reaches a tally of 100, allowing it to return earlier. + +From an operator perspective, they may not care for the exact value just that it's a value which is exceeding a given threshold. + +On a similar thought process, a "minimum frequency" could be supported. For instance, the API request could set a `min-frequency=100`. + +The server would only return metric/label/value matches if their associated frequency was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. + +With the combination of this min value and the `limit` it would allow the server to return more quickly. + +For instance, this could be used for searches such as "find me 10 metrics whose name contains `cpu` with a cardinality > 100". + +### Testing and verification + +It should be possible to validate these new endpoints via comparing to the existing endpoints. + +Queries to existing endpoints can be constructed to extract a full set of labels and values and this data can be client side processed and compared to the equivalent new endpoint responses. + +### Migration + +These are new endpoints and do not change or alter any existing functionality. No migration is required. + +### Known unknowns + +* confirm feasibility of including frequency and cardinality in these API responses +* confirm feasibility of supporting cursor based pagination for these new endpoints +* confirm any performance / response time constraints for these new endpoints +* specific choice of fuzzy search algorithm + +## Alternatives + +### 1. Can we not just use the existing Prometheus endpoints? + +For large scale environments with client-side optimizations; + +* High-cardinality scenarios remain challenging (43MB responses, timeouts) +* OpenTelemetry adoption increases frequency of these scenarios +* Frontend complexity grows (caching, throttling, workarounds) +* User experience compromises persist (limited search, arbitrary limits) + +### 2. Can we extend/enhance the existing Prometheus endpoints? + +The existing API design and response format is constrained. + +The existing response format returns collections of strings which does not support additional record enrichment. + +Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes. + +### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \__name__? + +Separating this endpoint allows for the additional enrichment options to be included in both the parameters and response. + +Although these could be applied to the values endpoint, it would result in the values endpoint have different behaviours (input parameters and response format) depending on the label name. + +A dedicated endpoint also allows for future enrichments to be added which are only relevant to metric names. + +## Action Plan + +The tasks to do in order to migrate to the new idea. + +* [ ] Task one `` +* [ ] Task two `` ... From 2b2de0089f2e18f37dc018720b1ec9cb980767f3 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 29 Jan 2026 14:52:57 +0800 Subject: [PATCH 02/22] Lint Signed-off-by: Andrew Hall --- ...s-api.md => 0074-new-labels-values-api.md} | 256 ++++++++++++------ 1 file changed, 177 insertions(+), 79 deletions(-) rename proposals/{0000-new-labels-values-api.md => 0074-new-labels-values-api.md} (69%) diff --git a/proposals/0000-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md similarity index 69% rename from proposals/0000-new-labels-values-api.md rename to proposals/0074-new-labels-values-api.md index 04c8c97..8b8bf91 100644 --- a/proposals/0000-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -3,7 +3,7 @@ * **Owners:** * Ismail Simsek [@itsmylife](https://github.com/itsmylife) [ismail.simsek@grafana.com](mailto:ismail.simsek@grafana.com) * Andrew Hall [@tcp13equals2](https://github.com/tcp13equals2) [andrew.hall@grafana.com](mailto:andrew.hall@grafana.com) - + * **Implementation Status:** Not implemented * **Related Issues and PRs:** @@ -23,9 +23,9 @@ The existing Prometheus [metadata API endpoints](https://github.com/prometheus/p These endpoints are heavily relied upon by the Grafana web application for core discovery workflows, including: -* __Autocomplete__ — triggered as users type queries -* __Metrics Explorer / Builder__ — a click-based interface for browsing metrics and label dimensions -* __Metrics Drilldown__ — dynamically generating metric panels by iterating over label values +* **Autocomplete** — triggered as users type queries +* **Metrics Explorer / Builder** — a click-based interface for browsing metrics and label dimensions +* **Metrics Drilldown** — dynamically generating metric panels by iterating over label values As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, these existing endpoints have increasingly become a limiting factor. @@ -38,23 +38,23 @@ Grafana has attempted to mitigate these limitations through a range of client-si Pitfalls of the current metadata endpoints; -* __Limited client-side filtering__: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. -* __Limited matching capabilities__: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). -* __Unordered `limit` results__: `limit` returns an arbitrary subset of values rather than returning the most relevant or most frequent values, which causes drilldown UIs to display incomplete or unrepresentative results. -* __No metadata about further results__: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). -* __No pagination/cursors__: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. -* __Fragmented calls for full context__: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. -* __No streaming / incremental delivery__: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and perceived latency. +* **Limited client-side filtering**: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. +* **Limited matching capabilities**: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). +* **Unordered `limit` results**: `limit` returns an arbitrary subset of values rather than returning the most relevant or most frequent values, which causes drilldown UIs to display incomplete or unrepresentative results. +* **No metadata about further results**: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). +* **No pagination/cursors**: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. +* **Fragmented calls for full context**: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. +* **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and perceived latency. ## Goals Effective data queries cannot be written if what needs to be queried cannot be identified first. Improving metric name, label name and label value discovery reduces Mean Time to Execute Query (MTEQ). This can be rephrased as “users spend less time hunting for the right metrics and more time analyzing data.” -* __Faster metric name and label identification__ - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" -* __Better autocomplete performance__ - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side -* __Scalable high-cardinality handling__ -server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion -* __Reduced bandwidth/resource consumption__ - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user -* __Improved AI/ML agent efficiency__ - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times +* **Faster metric name and label identification** - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" +* **Better autocomplete performance** - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side +* **Scalable high-cardinality handling** -server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion +* **Reduced bandwidth/resource consumption** - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user +* **Improved AI/ML agent efficiency** - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times ### Audience @@ -67,7 +67,7 @@ Effective data queries cannot be written if what needs to be queried cannot be i ## How -The following API endpoints are proposed. Each endpoint should return data in a NDJSON / chunked encoding streaming format. +The following API endpoints are proposed. Each endpoint should return data in a NDJSON / chunked encoding streaming format. ### `GET /api/v1/search/metrics` @@ -75,38 +75,79 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai #### Request -**Method:** `GET` `POST` +**Method:** `GET` `POST` + **Path:** `/api/v1/search/metrics` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------------|----------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `search` | string | No | N/A | The search string to be used for matching against metric names.

Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold.

The matching score of a fuzzy match must be >= this value.

A value of 100 disables any fuzz matching. | -| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | -| `sort_by` | alpha
cardinality
frequency
score | No | N/A | A flag to define how matching metrics should be sorted in the response.

* __alpha__ - metrics are sorted by alphabetical order
* __cardinality__ - metrics are sorted by their cardinality
* __frequency__ - metrics are sorted by their frequency of use
* __score__ - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. -| `sort_dir` | asc
dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. -| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. -| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. -| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. -| `start` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `end` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | -| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk.
A value of 0 indicates that the server can determine the batch size, which may be variable. | -| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | - | +| Name | Type | Required | Default | Description | +|----------|--------|----------|---------|-----------------------------------------------------------------| +| `search` | string | No | N/A | The search string to be used for matching against metric names. | +|
| | | | | + +
+Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold. +
+ +
+The matching score of a fuzzy match must be >= this value. +
+ +
+A value of 100 disables any fuzz matching. | +| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | +| `sort_by` | alpha +
+cardinality +
+frequency +
+score | No | N/A | A flag to define how matching metrics should be sorted in the response. +
+ +
+* __alpha__ - metrics are sorted by alphabetical order +
+* __cardinality__ - metrics are sorted by their cardinality +
+* __frequency__ - metrics are sorted by their frequency of use +
+* __score__ - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. +
+ +
+Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc +
+dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. | +| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. | +| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. | +| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. | +| `start` | rfc3339 +
+unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `end` | rfc3339 +
+unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | +| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. +
+A value of 0 indicates that the server can determine the batch size, which may be variable. | +| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | +| | | | | | **Notes:** - If `search` is omitted or empty, all metric names are matched. -- The fuzzy search could be a Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to a 0..100 threshold. A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100` +- The fuzzy search could be a Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to a 0..100 threshold. A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100` - Support for `cursor` based pagination is desirable but not essential. - `start` and `end` could have default values set which align to a reasonable look-back period. ie last 24 hours - `include_*` should use `json:"omitempty"` to not serialize the enriched attributes if they have not been requested / initialised Example fuzzy matching; -* Jaro-Winkler, mimir vs mimer = 0.953 +* Jaro-Winkler, mimir vs mimer = 0.953 * Levenshtein, mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8 #### Response @@ -229,31 +270,62 @@ As per existing endpoints. ### `GET /api/v1/search/labels` -An endpoint specific to searching for label names. +An endpoint specific to searching for label names. #### Request -**Method:** `GET` `POST` +**Method:** `GET` `POST` + **Path:** `/api/v1/search/labels` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | -| `search` | string | No | N/A | The search string to be used for matching against label names.

Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha
cardinality
frequency
score | No | N/A | A flag to define how matching labels should be sorted in the response.

* __alpha__ - labels are sorted by alphabetical order
* __cardinality__ - labels are sorted by their value cardinality
* __frequency__ - labels are sorted by their frequency of use
* __score__ - labels are sorted by a matching score.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. -| `sort_dir` | asc
dsc | No | asc | As per above endpoint -| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. -| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. -| `start` | rfc3339
unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339
unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | -| +| Name | Type | Required | Default | Description | +|-----------|--------------------|----------|---------|-------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | +| `search` | string | No | N/A | The search string to be used for matching against label names. | +|
| | | | | + +
+Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha +
+cardinality +
+frequency +
+score | No | N/A | A flag to define how matching labels should be sorted in the response. +
+ +
+* __alpha__ - labels are sorted by alphabetical order +
+* __cardinality__ - labels are sorted by their value cardinality +
+* __frequency__ - labels are sorted by their frequency of use +
+* __score__ - labels are sorted by a matching score. +
+ +
+Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc +
+dsc | No | asc | As per above endpoint | +| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. | +| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. | +| `start` | rfc3339 +
+unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339 +
+unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | +| | | | | | **Notes:** - If `search` is omitted or empty, all label names are matched. @@ -328,27 +400,54 @@ An endpoint specific to searching for label values. #### Request -**Method:** `GET` `POST` +**Method:** `GET` `POST` + **Path:** `/api/v1/search/values` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|------------------------------------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | -| `label` | string | Yes | N/A | The label the user is requesting values for. | -| `search` | string | No | N/A | The search string to be used for matching against label values.

Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha
frequency
score | No | N/A | A flag to define how matching values should be sorted in the response.

* __alpha__ - values are sorted by alphabetical order
* __frequency__ - values are sorted by their frequency of use
* __score__ - values are sorted by a matching score.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. -| `sort_dir` | asc
dsc | No | asc | As per above endpoint -| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. -| `start` | rfc3339
unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339
unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | -| +| Name | Type | Required | Default | Description | +|-----------|--------------------|----------|---------|------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | +| `label` | string | Yes | N/A | The label the user is requesting values for. | +| `search` | string | No | N/A | The search string to be used for matching against label values. | +|
| | | | | + +
+Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha +
+frequency +
+score | No | N/A | A flag to define how matching values should be sorted in the response. +
+ +
+* __alpha__ - values are sorted by alphabetical order +
+* __frequency__ - values are sorted by their frequency of use +
+* __score__ - values are sorted by a matching score. +
+ +
+Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc +
+dsc | No | asc | As per above endpoint | +| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. | +| `start` | rfc3339 +
+unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339 +
+unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | +| | | | | | **Notes:** - If `search` is omitted or empty, all label values are matched. @@ -366,7 +465,6 @@ Use existing Prometheus API status codes. ### Example of NDJSON batched result set - no include_* flags set - ```ndjson { "results": [ @@ -419,15 +517,15 @@ Use existing Prometheus API status codes. ### Additional notes -For the `frequency` and `cardinality` enrichments, an idea is to support a "maximum reported threshold". +For the `frequency` and `cardinality` enrichments, an idea is to support a "maximum reported threshold". -For instance, the API request could include a `max-reported-frequency=100`. As the server side indexes the frequency of use it can stop once it reaches a tally of 100, allowing it to return earlier. +For instance, the API request could include a `max-reported-frequency=100`. As the server side indexes the frequency of use it can stop once it reaches a tally of 100, allowing it to return earlier. From an operator perspective, they may not care for the exact value just that it's a value which is exceeding a given threshold. -On a similar thought process, a "minimum frequency" could be supported. For instance, the API request could set a `min-frequency=100`. +On a similar thought process, a "minimum frequency" could be supported. For instance, the API request could set a `min-frequency=100`. -The server would only return metric/label/value matches if their associated frequency was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. +The server would only return metric/label/value matches if their associated frequency was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. With the combination of this min value and the `limit` it would allow the server to return more quickly. @@ -435,7 +533,7 @@ For instance, this could be used for searches such as "find me 10 metrics whose ### Testing and verification -It should be possible to validate these new endpoints via comparing to the existing endpoints. +It should be possible to validate these new endpoints via comparing to the existing endpoints. Queries to existing endpoints can be constructed to extract a full set of labels and values and this data can be client side processed and compared to the equivalent new endpoint responses. @@ -465,11 +563,11 @@ For large scale environments with client-side optimizations; The existing API design and response format is constrained. -The existing response format returns collections of strings which does not support additional record enrichment. +The existing response format returns collections of strings which does not support additional record enrichment. Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes. -### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \__name__? +### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? Separating this endpoint allows for the additional enrichment options to be included in both the parameters and response. From e63410d216f61055bed112d189ae57b49c6f9a7a Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 30 Jan 2026 09:31:12 +0800 Subject: [PATCH 03/22] Lint Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 8b8bf91..f114e64 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -84,7 +84,9 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai | Name | Type | Required | Default | Description | |----------|--------|----------|---------|-----------------------------------------------------------------| | `search` | string | No | N/A | The search string to be used for matching against metric names. | +| | | | | | |
| | | | | +| | | | | |
Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | @@ -92,7 +94,7 @@ Metric names are matched if they contain this search string or a fuzzy match mee

-The matching score of a fuzzy match must be >= this value. +The matching score of a fuzzy match must be >= this value.

@@ -131,8 +133,8 @@ unix_timestamp | No | | As per the current labe | `end` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | -| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. +| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | +| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk.
A value of 0 indicates that the server can determine the batch size, which may be variable. | | `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | @@ -284,7 +286,9 @@ An endpoint specific to searching for label names. |-----------|--------------------|----------|---------|-------------------------------------------------------------------------------------------------------------| | `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | | `search` | string | No | N/A | The search string to be used for matching against label names. | +| | | | | | |
| | | | | +| | | | | |
Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | @@ -411,7 +415,9 @@ An endpoint specific to searching for label values. | `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | | `label` | string | Yes | N/A | The label the user is requesting values for. | | `search` | string | No | N/A | The search string to be used for matching against label values. | +| | | | | | |
| | | | | +| | | | | |
Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | From a0bc2290244a772833ded61ea9ef761de6cbe218 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 30 Jan 2026 09:45:23 +0800 Subject: [PATCH 04/22] Lint Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 195 ++++++------------------ 1 file changed, 45 insertions(+), 150 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index f114e64..2522c4e 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -81,64 +81,21 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai **Query parameters:** -| Name | Type | Required | Default | Description | -|----------|--------|----------|---------|-----------------------------------------------------------------| -| `search` | string | No | N/A | The search string to be used for matching against metric names. | -| | | | | | -|
| | | | | -| | | | | | - -
-Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold. -
- -
-The matching score of a fuzzy match must be >= this value. -
- -
-A value of 100 disables any fuzz matching. | -| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | -| `sort_by` | alpha -
-cardinality -
-frequency -
-score | No | N/A | A flag to define how matching metrics should be sorted in the response. -
- -
-* __alpha__ - metrics are sorted by alphabetical order -
-* __cardinality__ - metrics are sorted by their cardinality -
-* __frequency__ - metrics are sorted by their frequency of use -
-* __score__ - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. -
- -
-Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc -
-dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. | -| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. | -| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. | -| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. | -| `start` | rfc3339 -
-unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `end` | rfc3339 -
-unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | -| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. -
-A value of 0 indicates that the server can determine the batch size, which may be variable. | -| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | -| | | | | | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `search` | string | No | N/A | The search string to be used for matching against metric names. Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold. The matching score of a fuzzy match must be >= this value. A value of 100 disables any fuzz matching. | +| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | +| `sort_by` | alpha / cardinality / frequency / score | No | N/A | A flag to define how matching metrics should be sorted in the response. **alpha** - metrics are sorted by alphabetical order. **cardinality** - metrics are sorted by their cardinality. **frequency** - metrics are sorted by their frequency of use. **score** - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc / dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. | +| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. | +| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. | +| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. | +| `start` | rfc3339 / unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `end` | rfc3339 / unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | +| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. A value of 0 indicates that the server can determine the batch size, which may be variable. | +| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | **Notes:** - If `search` is omitted or empty, all metric names are matched. @@ -282,54 +239,21 @@ An endpoint specific to searching for label names. **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------|--------------------|----------|---------|-------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | -| `search` | string | No | N/A | The search string to be used for matching against label names. | -| | | | | | -|
| | | | | -| | | | | | - -
-Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha -
-cardinality -
-frequency -
-score | No | N/A | A flag to define how matching labels should be sorted in the response. -
- -
-* __alpha__ - labels are sorted by alphabetical order -
-* __cardinality__ - labels are sorted by their value cardinality -
-* __frequency__ - labels are sorted by their frequency of use -
-* __score__ - labels are sorted by a matching score. -
- -
-Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc -
-dsc | No | asc | As per above endpoint | -| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. | -| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. | -| `start` | rfc3339 -
-unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339 -
-unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | -| | | | | | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | +| `search` | string | No | N/A | The search string to be used for matching against label names. Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha / cardinality / frequency / score | No | N/A | A flag to define how matching labels should be sorted in the response. **alpha** - labels are sorted by alphabetical order. **cardinality** - labels are sorted by their value cardinality. **frequency** - labels are sorted by their frequency of use. **score** - labels are sorted by a matching score. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint | +| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. | +| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | **Notes:** - If `search` is omitted or empty, all label names are matched. @@ -410,50 +334,21 @@ An endpoint specific to searching for label values. **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------|--------------------|----------|---------|------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | -| `label` | string | Yes | N/A | The label the user is requesting values for. | -| `search` | string | No | N/A | The search string to be used for matching against label values. | -| | | | | | -|
| | | | | -| | | | | | - -
-Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha -
-frequency -
-score | No | N/A | A flag to define how matching values should be sorted in the response. -
- -
-* __alpha__ - values are sorted by alphabetical order -
-* __frequency__ - values are sorted by their frequency of use -
-* __score__ - values are sorted by a matching score. -
- -
-Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc -
-dsc | No | asc | As per above endpoint | -| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. | -| `start` | rfc3339 -
-unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339 -
-unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | -| | | | | | +| Name | Type | Required | Default | Description | +|---------------------|---------------------------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | +| `label` | string | Yes | N/A | The label the user is requesting values for. | +| `search` | string | No | N/A | The search string to be used for matching against label values. Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha / frequency / score | No | N/A | A flag to define how matching values should be sorted in the response. **alpha** - values are sorted by alphabetical order. **frequency** - values are sorted by their frequency of use. **score** - values are sorted by a matching score. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint | +| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | **Notes:** - If `search` is omitted or empty, all label values are matched. From 22e3123b3f2982011f8b4bb2b41ae9990b3cf38f Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 30 Jan 2026 12:13:34 +0800 Subject: [PATCH 05/22] Lint Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 215 +++++++++++++++--------- 1 file changed, 136 insertions(+), 79 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 2522c4e..2f1fb23 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -1,8 +1,8 @@ ## V2 API for labels and values discovery * **Owners:** - * Ismail Simsek [@itsmylife](https://github.com/itsmylife) [ismail.simsek@grafana.com](mailto:ismail.simsek@grafana.com) - * Andrew Hall [@tcp13equals2](https://github.com/tcp13equals2) [andrew.hall@grafana.com](mailto:andrew.hall@grafana.com) + * Ismail Simsek [@itsmylife](https://github.com/itsmylife) ismail.simsek@grafana.com + * Andrew Hall [@tcp13equals2](https://github.com/tcp13equals2) andrew.hall@grafana.com * **Implementation Status:** Not implemented @@ -27,16 +27,16 @@ These endpoints are heavily relied upon by the Grafana web application for core * **Metrics Explorer / Builder** — a click-based interface for browsing metrics and label dimensions * **Metrics Drilldown** — dynamically generating metric panels by iterating over label values -As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, these existing endpoints have increasingly become a limiting factor. +As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, the existing Prometheus endpoints have increasingly become a limiting factor. -In practice, this manifests as performance bottlenecks in the Grafana UI and constrains the implementation of user-requested discovery features, particularly those that require iterative or interactive exploration of large metadata sets. +This manifests as performance bottlenecks in the Grafana UI and constrains the implementation of user-requested discovery features, particularly those that require iterative or interactive exploration of large metadata sets. Grafana has attempted to mitigate these limitations through a range of client-side optimizations and workarounds (see [#100376](https://github.com/grafana/grafana/issues/100376) ). However, these approaches add complexity and cannot fully address the underlying scalability and efficiency constraints of the current metadata APIs. ### Pitfalls of the current solution -Pitfalls of the current metadata endpoints; +Pitfalls of the current metadata endpoints: * **Limited client-side filtering**: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. * **Limited matching capabilities**: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). @@ -44,7 +44,7 @@ Pitfalls of the current metadata endpoints; * **No metadata about further results**: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). * **No pagination/cursors**: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. * **Fragmented calls for full context**: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. -* **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and perceived latency. +* **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and low latency interactions. ## Goals @@ -52,62 +52,113 @@ Effective data queries cannot be written if what needs to be queried cannot be i * **Faster metric name and label identification** - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" * **Better autocomplete performance** - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side -* **Scalable high-cardinality handling** -server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion +* **Scalable high-cardinality handling** - server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion * **Reduced bandwidth/resource consumption** - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user * **Improved AI/ML agent efficiency** - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times ### Audience -* Operators seeking to discover metrics, labels and values +* Operators seeking to discover metrics, labels and values - operators seeking to create new queries, perform adhoc investigations, create new rule/alerts, cardinality exploration etc ## Non-Goals -* Changing/enhancing/deprecating any existing Prometheus API endpoints related to labels, values and metadata -* Addressing OpenTelemetry hidden label limitations. Resource attributes stored in `target_info` rather than on every series and therefor not surfaced in the existing labels' endpoint. +* Deprecating / replacing any existing Prometheus API endpoints +* Addressing OpenTelemetry hidden label limitations. Resource attributes stored in `target_info` rather than on every series and therefore not surfaced in the existing labels' endpoint. ## How -The following API endpoints are proposed. Each endpoint should return data in a NDJSON / chunked encoding streaming format. +A new set of `search` endpoints are proposed. -### `GET /api/v1/search/metrics` +* /api/v1/search/metric_names +* /api/v1/search/label_names +* /api/v1/search/label_values -An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric. +Each endpoint allows for the specific searching, filtering, sorting of metric names, label names and label values respectively. + +Note that; + +* the existing Prometheus API labels/values parameter set is re-used in each endpoint, with additional parameters introduced to deliver the desired functionality +* the response format allows for enriched data to be added to each metric name, label name or label value record - rather than just a collection of strings +* the response is NDJSON (`application/x-ndjson`), allowing for a streamed chunked encoding response + +### `GET /api/v1/search/metric_names` + +An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric name. #### Request **Method:** `GET` `POST` -**Path:** `/api/v1/search/metrics` +**Path:** `/api/v1/search/metric_names` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `search` | string | No | N/A | The search string to be used for matching against metric names. Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold. The matching score of a fuzzy match must be >= this value. A value of 100 disables any fuzz matching. | -| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | -| `sort_by` | alpha / cardinality / frequency / score | No | N/A | A flag to define how matching metrics should be sorted in the response. **alpha** - metrics are sorted by alphabetical order. **cardinality** - metrics are sorted by their cardinality. **frequency** - metrics are sorted by their frequency of use. **score** - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc / dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. | -| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. | -| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. | -| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. | -| `start` | rfc3339 / unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `end` | rfc3339 / unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | -| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. A value of 0 indicates that the server can determine the batch size, which may be variable. | -| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|------------------------------------------------------------------------------| +| `search` | string | No | | The search string to be used for matching metric names. | +| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | +| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | +| `sort_by` | alpha / cardinality / frequency / score | No | | Request how matching metrics should be sorted in the response. | +| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | +| `include_frequency` | bool | No | false | Request metric frequency. | +| `include_cardinality` | bool | No | false | Request metric cardinality. | +| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | +| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | +| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | +| `cursor` | string | No | | Request the next page of results. | **Notes:** -- If `search` is omitted or empty, all metric names are matched. -- The fuzzy search could be a Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to a 0..100 threshold. A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100` -- Support for `cursor` based pagination is desirable but not essential. -- `start` and `end` could have default values set which align to a reasonable look-back period. ie last 24 hours -- `include_*` should use `json:"omitempty"` to not serialize the enriched attributes if they have not been requested / initialised -Example fuzzy matching; +***search*** + +The given `search` value will be used to match metric names. + +A match is found if the metric name contains this search value or if a fuzzy match between the metric name and this search meets the given `fuzz_threshold`. + +If `search` is omitted or empty, all metric names are matched. + +***fuzz_threshold*** + +The fuzz matching score must meet or exceed this threshold to be considered a match. + +A value of 100 disables any fuzz matching. + +The fuzzy search could be a Jaro-Winkler or Levenshtein match. + +A Jaro-Winkler returns a score between 0..1 which can be easily scaled to 0..100. mimir vs mimer = 0.953. A `fuzz_threshold` of 95 or below would allow this match. + +A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100`. mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8. A `fuzz_threshold` of 80 or below would allow this match. + +***sort_by*** + +* **alpha** - metric names are sorted by alphabetical order. +* **cardinality** - metric names are sorted by their cardinality. +* **frequency** - metric names are sorted by their frequency of use. +* **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. -* Jaro-Winkler, mimir vs mimer = 0.953 -* Levenshtein, mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8 +Note that `sort_by` is optional, and it is valid for there to be no sorting requested. This allows the server to return / stream results back immediately without the need for any server-side buffering. + +***batch_size*** + +The desired size of each batch of results sent in each response chunk. + +A value of 0 indicates that the server can determine the batch size, which may be variable. + +***cursor*** + +Note that cursor based pagination is desirable but not essential. + +Note that this parameter can only be used with the `limit` parameter. The request should fail if the `cursor` parameter is set with any other parameter. + +***start/end*** + +It is proposed that these could have default values which align to a reasonable look-back period. ie last 24 hours + +***include_*** + +The implementation of `include_*` should use `json:"omitempty"` to not serialize the enriched attributes if they have not been requested / initialised #### Response @@ -227,7 +278,7 @@ As per existing endpoints. } ``` -### `GET /api/v1/search/labels` +### `GET /api/v1/search/label_names` An endpoint specific to searching for label names. @@ -235,28 +286,34 @@ An endpoint specific to searching for label names. **Method:** `GET` `POST` -**Path:** `/api/v1/search/labels` +**Path:** `/api/v1/search/label_names` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | -| `search` | string | No | N/A | The search string to be used for matching against label names. Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha / cardinality / frequency / score | No | N/A | A flag to define how matching labels should be sorted in the response. **alpha** - labels are sorted by alphabetical order. **cardinality** - labels are sorted by their value cardinality. **frequency** - labels are sorted by their frequency of use. **score** - labels are sorted by a matching score. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint | -| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. | -| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | string | No | | The search string to be used for matching against label names. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / cardinality / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request label frequency. | +| `include_cardinality` | bool | No | false | Request label cardinality. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | +| `cursor` | string | No | | As per above endpoint. | **Notes:** -- If `search` is omitted or empty, all label names are matched. + +***sort_by*** + +* **alpha** - label names are sorted by alphabetical order. +* **cardinality** - label names are sorted by their cardinality. +* **frequency** - label names are sorted by their frequency of use. +* **score** - label names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. #### Response @@ -322,7 +379,7 @@ Use existing Prometheus API status codes. } ``` -### `GET /api/v1/search/values` +### `GET /api/v1/search/label_values` An endpoint specific to searching for label values. @@ -330,29 +387,28 @@ An endpoint specific to searching for label values. **Method:** `GET` `POST` -**Path:** `/api/v1/search/values` +**Path:** `/api/v1/search/label_values` **Query parameters:** -| Name | Type | Required | Default | Description | -|---------------------|---------------------------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | -| `label` | string | Yes | N/A | The label the user is requesting values for. | -| `search` | string | No | N/A | The search string to be used for matching against label values. Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha / frequency / score | No | N/A | A flag to define how matching values should be sorted in the response. **alpha** - values are sorted by alphabetical order. **frequency** - values are sorted by their frequency of use. **score** - values are sorted by a matching score. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint | -| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | +| Name | Type | Required | Default | Description | +|---------------------|---------------------------|----------|---------|-----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `label` | string | Yes | | The label the user is requesting values for. | +| `search` | string | No | | The search string to be used for matching against label values. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request value frequency. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | +| `cursor` | string | No | | As per above endpoint. | **Notes:** -- If `search` is omitted or empty, all label values are matched. -- The `label` parameter has been deliberately added as a required query parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. +- The `label` parameter has been deliberately added as a required `query` parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. #### Response @@ -364,7 +420,7 @@ Use existing Prometheus API status codes. * Content-Type: application/x-ndjson; charset=utf-8 -### Example of NDJSON batched result set - no include_* flags set +##### Example of NDJSON batched result set - no include_* flags set ```ndjson { @@ -388,7 +444,7 @@ Use existing Prometheus API status codes. } ``` -### Example of NDJSON batched result set - with include_* flags set +##### Example of NDJSON batched result set - with include_* flags set ```ndjson { @@ -398,7 +454,7 @@ Use existing Prometheus API status codes. "frequency": 1003 }, { - "name": "cluster1", + "name": "cluster2", "frequency": 4 } ] @@ -448,12 +504,13 @@ These are new endpoints and do not change or alter any existing functionality. N * confirm feasibility of supporting cursor based pagination for these new endpoints * confirm any performance / response time constraints for these new endpoints * specific choice of fuzzy search algorithm +* specific implementation of the search result ordering for auto-complete scenarios ## Alternatives ### 1. Can we not just use the existing Prometheus endpoints? -For large scale environments with client-side optimizations; +For large scale environments even with client-side optimizations; * High-cardinality scenarios remain challenging (43MB responses, timeouts) * OpenTelemetry adoption increases frequency of these scenarios @@ -466,7 +523,7 @@ The existing API design and response format is constrained. The existing response format returns collections of strings which does not support additional record enrichment. -Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes. +Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes to these endpoints. ### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? From 0cec59be645bb4ff25b9bbee00b2a33c1feef4bd Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:36:41 +0800 Subject: [PATCH 06/22] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 2f1fb23..0dd2e71 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -87,7 +87,9 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai #### Request -**Method:** `GET` `POST` +**Method:** `GET` or `POST` + +`POST` is recommended when query parameters may exceed URL length limits. **Path:** `/api/v1/search/metric_names` From 22a01618fe084f484aa0cfdbf6c7c7938c943dc8 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:37:05 +0800 Subject: [PATCH 07/22] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 0dd2e71..fd696f1 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -137,7 +137,8 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance * **alpha** - metric names are sorted by alphabetical order. * **cardinality** - metric names are sorted by their cardinality. -* **frequency** - metric names are sorted by their frequency of use. +* **frequency** - metric names are sorted by their frequency of use (i.e., the number of active series containing this metric name within the queried time range). +* ``` * **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that `sort_by` is optional, and it is valid for there to be no sorting requested. This allows the server to return / stream results back immediately without the need for any server-side buffering. From 60075c996c2d858308b6f0b7835ea9ca3221faf9 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:37:25 +0800 Subject: [PATCH 08/22] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index fd696f1..94d81cf 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -287,7 +287,9 @@ An endpoint specific to searching for label names. #### Request -**Method:** `GET` `POST` +**Method:** `GET` or `POST` + +`POST` is recommended when `match[]` selectors may exceed URL length limits. **Path:** `/api/v1/search/label_names` From e16b4d8a103434f5ff187ec9c73aa6e8872f7baf Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:37:37 +0800 Subject: [PATCH 09/22] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 94d81cf..ba9504b 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -317,7 +317,7 @@ An endpoint specific to searching for label names. * **alpha** - label names are sorted by alphabetical order. * **cardinality** - label names are sorted by their cardinality. -* **frequency** - label names are sorted by their frequency of use. +* **frequency** - label names are sorted by their frequency of use (i.e., the number of active series containing this label within the queried time range). * **score** - label names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. #### Response From d80aa080dc2fc71824fce95cffbc3bb1c474d41b Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:37:48 +0800 Subject: [PATCH 10/22] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index ba9504b..207b645 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -390,7 +390,9 @@ An endpoint specific to searching for label values. #### Request -**Method:** `GET` `POST` +**Method:** `GET` or `POST` + +`POST` is recommended when `match[]` selectors may exceed URL length limits. **Path:** `/api/v1/search/label_values` From be972af903fc91ea86a7487647b384803e6ea7db Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:38:12 +0800 Subject: [PATCH 11/22] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 207b645..d412670 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -544,5 +544,10 @@ A dedicated endpoint also allows for future enrichments to be added which are on The tasks to do in order to migrate to the new idea. -* [ ] Task one `` -* [ ] Task two `` ... +* [ ] Finalize proposal based on community feedback +* [ ] Create Prometheus implementation issue for `/api/v1/search/metric_names` +* [ ] Create Prometheus implementation issue for `/api/v1/search/label_names` +* [ ] Create Prometheus implementation issue for `/api/v1/search/label_values` +* [ ] Implement endpoints in Prometheus +* [ ] Integrate new endpoints into Prometheus UI +* [ ] Update Prometheus documentation From 73ea349c4b314945d582647588daee0f51937696 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:53:49 +0800 Subject: [PATCH 12/22] Update 0074-new-labels-values-api.md Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index d412670..1b4e15b 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -97,6 +97,7 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai | Name | Type | Required | Default | Description | |-----------------------|-----------------------------------------|----------|---------|------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | | `search` | string | No | | The search string to be used for matching metric names. | | `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | | `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | @@ -121,6 +122,12 @@ A match is found if the metric name contains this search value or if a fuzzy mat If `search` is omitted or empty, all metric names are matched. +The `match[]` param can be used as an alternative to using the `search` or it can be used in conjunction with the `search`. + +For example; +* `match[]={cluster=prod}&search=` - find me all metric names which have a `cluster=prod` label +* `match[]={cluster=prod}&search=cpu` - find me all metric names which contain `cpu` and which have a `cluster=prod` label + ***fuzz_threshold*** The fuzz matching score must meet or exceed this threshold to be considered a match. From 03850cb705640ac8fd6639b535b41a1666b92d2e Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 12:07:31 +0800 Subject: [PATCH 13/22] PR feedback Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 1b4e15b..a7b2870 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -36,7 +36,18 @@ Grafana has attempted to mitigate these limitations through a range of client-si ### Pitfalls of the current solution -Pitfalls of the current metadata endpoints: +The current metadata APIs provide limited server-side filtering, ordering, and aggregation capabilities. As a result, client applications are often required to retrieve large, unfiltered datasets and then perform filtering, sorting, and relevance ranking locally. + +In addition, clients must frequently combine data from multiple API endpoints to assemble the information needed for a single user interaction or view. + +In practice, this leads to: +* unnecessarily **large response payloads** +* **increased backend load** +* a degraded user experience, as **clients must wait** for full responses before rendering meaningful results + +For interactive discovery workflows, this significantly increases the time-to-first-result presented to the user and constrains the ability to build responsive, incremental interfaces. + +Some specific examples highlighted by Grafana web application developers include; * **Limited client-side filtering**: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. * **Limited matching capabilities**: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). From 24c9b9d273cf1dfa28914e2ca28457550fd96c54 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 12:44:06 +0800 Subject: [PATCH 14/22] PR feedback Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index a7b2870..6b3259f 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -157,6 +157,7 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance * **cardinality** - metric names are sorted by their cardinality. * **frequency** - metric names are sorted by their frequency of use (i.e., the number of active series containing this metric name within the queried time range). * ``` + ``` * **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that `sort_by` is optional, and it is valid for there to be no sorting requested. This allows the server to return / stream results back immediately without the need for any server-side buffering. From c114ce460b5ef68f2e1080f2b60f5382d45b6f4b Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 5 Feb 2026 08:17:33 +0800 Subject: [PATCH 15/22] Rename cursor param to next_token Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 6b3259f..64214ea 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -121,7 +121,7 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai | `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | | `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | | `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | -| `cursor` | string | No | | Request the next page of results. | +| `next_token` | string | No | | Request the next page of results. | **Notes:** @@ -168,11 +168,11 @@ The desired size of each batch of results sent in each response chunk. A value of 0 indicates that the server can determine the batch size, which may be variable. -***cursor*** +***next_token*** -Note that cursor based pagination is desirable but not essential. +Note that pagination is desirable but not essential to this proposal. -Note that this parameter can only be used with the `limit` parameter. The request should fail if the `cursor` parameter is set with any other parameter. +Note that this parameter can only be used with the `limit` parameter. The request should fail if the `next_token` parameter is set with any other parameter. ***start/end*** @@ -254,7 +254,7 @@ Use existing Prometheus API status codes. ##### Example of pagination token (if supported) -If cursor based pagination is supported, the `has_more` attribute is not required. The presence of the `cursor` attribute in the response indicates that there is more results available. +If pagination is supported, the `has_more` attribute is not required. The presence of the `next_token` attribute in the response indicates that there is more results available. ```ndjson { @@ -271,7 +271,7 @@ If cursor based pagination is supported, the `has_more` attribute is not require { "status": "success", - "cursor": "" + "next_token": "" } ``` @@ -328,7 +328,7 @@ An endpoint specific to searching for label names. | `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | | `limit` | int >= 0 | No | 100 | As per above endpoint. | | `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `cursor` | string | No | | As per above endpoint. | +| `next_token` | string | No | | As per above endpoint. | **Notes:** @@ -431,7 +431,7 @@ An endpoint specific to searching for label values. | `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | | `limit` | int >= 0 | No | 100 | As per above endpoint. | | `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `cursor` | string | No | | As per above endpoint. | +| `next_token` | string | No | | As per above endpoint. | **Notes:** - The `label` parameter has been deliberately added as a required `query` parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. From 8d9929841d3bf76380509f84b2f0e7a93f47f014 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 5 Feb 2026 09:04:07 +0800 Subject: [PATCH 16/22] Add extra section about frequency and cardinality Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 36 +++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 64214ea..b883bd0 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -27,7 +27,7 @@ These endpoints are heavily relied upon by the Grafana web application for core * **Metrics Explorer / Builder** — a click-based interface for browsing metrics and label dimensions * **Metrics Drilldown** — dynamically generating metric panels by iterating over label values -As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, the existing Prometheus endpoints have increasingly become a limiting factor. +As environments have grown in series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, the existing Prometheus endpoints have increasingly become a limiting factor. This manifests as performance bottlenecks in the Grafana UI and constrains the implementation of user-requested discovery features, particularly those that require iterative or interactive exploration of large metadata sets. @@ -125,6 +125,38 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai **Notes:** +***frequency & cardinality*** + +For the purposes of this document, the following definitions apply; + +* **frequency** — number of occurrences across a dataset within the queried time range. +* **cardinality** — the number of distinct time series within the queried time range; equivalently, the number of unique label-value combinations for a given metric or label. + +For example consider these samples; + +```text +http_requests_total{method="GET", status="200", instance="a"} +http_requests_total{method="GET", status="200", instance="b"} +http_requests_total{method="POST", status="200", instance="a"} +http_requests_total{method="POST", status="500", instance="a"} +http_requests_total{method="POST", status="500", instance="b"} + +cpu_usage_seconds_total{core="0", instance="a"} +cpu_usage_seconds_total{core="1", instance="a"} +cpu_usage_seconds_total{core="0", instance="b"} +cpu_usage_seconds_total{core="0", instance="b"} +``` + +| Name | Type | Frequency | Cardinality | +|-----------------------------------|-------------|-----------|-------------| +| http_requests_total | metric_name | 5 | 5 | +| http_requests_total{method="GET"} | metric_name | 2 | 2 | +| cpu_usage_seconds_total | metric_name | 4 | 3 | +| method | label_name | 5 | 2 | +| core | label_name | 4 | 2 | +| method="GET" | label_value | 2 | N/A | +| GET | label_value | 2 | N/A | + ***search*** The given `search` value will be used to match metric names. @@ -155,7 +187,7 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance * **alpha** - metric names are sorted by alphabetical order. * **cardinality** - metric names are sorted by their cardinality. -* **frequency** - metric names are sorted by their frequency of use (i.e., the number of active series containing this metric name within the queried time range). +* **frequency** - metric names are sorted by their frequency. * ``` ``` * **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. From 9f43fddf35e538d90bbe75869a8a09b065094aa1 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 5 Feb 2026 09:38:50 +0800 Subject: [PATCH 17/22] Add section for API extensibility Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 58 ++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index b883bd0..e60efd4 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -546,6 +546,62 @@ With the combination of this min value and the `limit` it would allow the server For instance, this could be used for searches such as "find me 10 metrics whose name contains `cpu` with a cardinality > 100". +### Extensibility for Mimir, Thanos, Cortex + +The introduction of returning collections of records rather than collections of strings allows for different implementations to provide additional record decorations. + +This would allow each implementation to provide a custom set of additional request parameters which results in additional data in the response. + +To avoid conflicts, the response record format could be extended to include an `extensions` map which returns the custom fields. The `extensions` would be omitted completely in the pure Prometheus implementation. + +```json +{ + "results": [ + { + "name": "cluster1", + "frequency": 1003, + "extensions": { + "mimir": { + "active_series": 10, + "owned_series" : 5, + ... + } + } + }, + { + "name": "cluster2", + "frequency": 4, + "extensions": { + ... + } + } + ] +} +``` + +The request to enable the extension could be implicit in the requested params. For instance the params could be `mimir.include_active_series=true`. + +To be more standards compliant, the extension could be requested in via the `Accept` header. ie `Accept: application/x-ndjson; charset=utf-8; extensions=mimir`. This would also be reflected in the returned `Content-Type`. + +If required for client side applications, a simple endpoint which returns the supported extensions and their data model could be exposed. + +*GET /api/v1/search/extensions* + +```json +{ + "mimir": { + "active_series": { + "type": "int", + "help": "some string explaining the field" + }, + ... + } +} + +``` + +If required, these extensions could also be versioned. + ### Testing and verification It should be possible to validate these new endpoints via comparing to the existing endpoints. @@ -583,7 +639,7 @@ The existing response format returns collections of strings which does not suppo Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes to these endpoints. -### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? +### 3. Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? Separating this endpoint allows for the additional enrichment options to be included in both the parameters and response. From 1a36be4f84598890eb04aa9a94774ec08795ba1f Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 08:48:49 +0800 Subject: [PATCH 18/22] Move pagination into feature request rather then limitation Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index e60efd4..8cf68c1 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -53,10 +53,11 @@ Some specific examples highlighted by Grafana web application developers include * **Limited matching capabilities**: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). * **Unordered `limit` results**: `limit` returns an arbitrary subset of values rather than returning the most relevant or most frequent values, which causes drilldown UIs to display incomplete or unrepresentative results. * **No metadata about further results**: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). -* **No pagination/cursors**: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. * **Fragmented calls for full context**: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. * **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and low latency interactions. +It is also noted that as part of this proposal there is also a **feature** request from the web application developers to support **pagination** (cursor/offset). *A request has been mde to the web application developers to confirm if this is needed assuming batched streaming of responses are supported.* + ## Goals Effective data queries cannot be written if what needs to be queried cannot be identified first. Improving metric name, label name and label value discovery reduces Mean Time to Execute Query (MTEQ). This can be rephrased as “users spend less time hunting for the right metrics and more time analyzing data.” From a29eb105d4a06c55d73249846b4bef9408be2158 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 09:15:20 +0800 Subject: [PATCH 19/22] Add warning example Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 73 ++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 3 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 8cf68c1..bcab7e0 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -89,9 +89,10 @@ Each endpoint allows for the specific searching, filtering, sorting of metric na Note that; -* the existing Prometheus API labels/values parameter set is re-used in each endpoint, with additional parameters introduced to deliver the desired functionality -* the response format allows for enriched data to be added to each metric name, label name or label value record - rather than just a collection of strings -* the response is NDJSON (`application/x-ndjson`), allowing for a streamed chunked encoding response +* the existing Prometheus API labels/values parameter set is re-used in each endpoint, with additional parameters introduced to deliver the desired functionality. +* the response format allows for enriched data to be added to each metric name, label name or label value record - rather than just a collection of strings. +* the response is NDJSON (`application/x-ndjson`), allowing for a streamed chunked encoding response. +* the existing Prometheus API response status, errors, warnings messages should be maintained. Note their placement in the response has been adapted for the streaming response format. ### `GET /api/v1/search/metric_names` @@ -333,6 +334,72 @@ As per existing endpoints. } ``` +##### Example of has_more + +Note - The `has_more` flag could replace the need for the existing Prometheus `results truncated due to limit` warning; + +```ndjson +{ + "results": [ + { + "name": "activity_tracker_failed_total", + }, + { + "name": "activity_tracker_free_slots", + } + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": true +} +``` + +##### Example of warnings + +Warnings can be added to either to an individual batch or to the overall request. + +As per the existing labels/values endpoints the `warnings` attribute is omitted in the response if there are none. + +If informational messages are also required they could be added in a similar way to a batch or the end message. + +```ndjson +{ + "results": [ + { + "name": "activity_tracker_failed_total", + }, + { + "name": "activity_tracker_free_slots", + } + ], + "warnings": [ + "some_warning_relevant_to_this_batch" + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": true, + "warnings": [ + "some_other_warning_relevant_to_the_overall_request" + ] +} +``` + ### `GET /api/v1/search/label_names` An endpoint specific to searching for label names. From 74b50ce8a95c810c55e22b6330dd94b9e1f49715 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 09:45:07 +0800 Subject: [PATCH 20/22] Fix use of frequency Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 101 ++++++++++++------------ 1 file changed, 52 insertions(+), 49 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index bcab7e0..dfbb5c8 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -87,52 +87,29 @@ A new set of `search` endpoints are proposed. Each endpoint allows for the specific searching, filtering, sorting of metric names, label names and label values respectively. -Note that; +### Implementation notes * the existing Prometheus API labels/values parameter set is re-used in each endpoint, with additional parameters introduced to deliver the desired functionality. * the response format allows for enriched data to be added to each metric name, label name or label value record - rather than just a collection of strings. * the response is NDJSON (`application/x-ndjson`), allowing for a streamed chunked encoding response. * the existing Prometheus API response status, errors, warnings messages should be maintained. Note their placement in the response has been adapted for the streaming response format. -### `GET /api/v1/search/metric_names` - -An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric name. - -#### Request - -**Method:** `GET` or `POST` - -`POST` is recommended when query parameters may exceed URL length limits. +#### frequency & cardinality -**Path:** `/api/v1/search/metric_names` +For the purposes of this document, the following definitions apply; -**Query parameters:** +* **frequency** — the number of occurrences of a label, or label value across time series within the queried time range. A single time series contributes at most one occurrence. +* **cardinality** — the number of distinct time series within the queried time range; equivalently, the number of unique label-value combinations for a given metric or label. -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|---------|------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `search` | string | No | | The search string to be used for matching metric names. | -| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | -| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | -| `sort_by` | alpha / cardinality / frequency / score | No | | Request how matching metrics should be sorted in the response. | -| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | -| `include_frequency` | bool | No | false | Request metric frequency. | -| `include_cardinality` | bool | No | false | Request metric cardinality. | -| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | -| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | -| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | -| `next_token` | string | No | | Request the next page of results. | +Cardinality answers "How many unique series exist?". -**Notes:** +Frequency answers "How commonly does this label / value appear across series?" -***frequency & cardinality*** +For a metric name, cardinality and frequency are equivalent so only cardinality is considered. -For the purposes of this document, the following definitions apply; +For a label name, cardinality and frequency are independent. -* **frequency** — number of occurrences across a dataset within the queried time range. -* **cardinality** — the number of distinct time series within the queried time range; equivalently, the number of unique label-value combinations for a given metric or label. +For a label value, only frequency is relevant. For example consider these samples; @@ -146,19 +123,49 @@ http_requests_total{method="POST", status="500", instance="b"} cpu_usage_seconds_total{core="0", instance="a"} cpu_usage_seconds_total{core="1", instance="a"} cpu_usage_seconds_total{core="0", instance="b"} -cpu_usage_seconds_total{core="0", instance="b"} ``` | Name | Type | Frequency | Cardinality | |-----------------------------------|-------------|-----------|-------------| -| http_requests_total | metric_name | 5 | 5 | -| http_requests_total{method="GET"} | metric_name | 2 | 2 | -| cpu_usage_seconds_total | metric_name | 4 | 3 | +| http_requests_total | metric_name | N/A | 5 | +| http_requests_total{method="GET"} | metric_name | N/A | 2 | | method | label_name | 5 | 2 | | core | label_name | 4 | 2 | | method="GET" | label_value | 2 | N/A | | GET | label_value | 2 | N/A | +### `GET /api/v1/search/metric_names` + +An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric name. + +#### Request + +**Method:** `GET` or `POST` + +`POST` is recommended when query parameters may exceed URL length limits. + +**Path:** `/api/v1/search/metric_names` + +**Query parameters:** + +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------|----------|---------|------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | string | No | | The search string to be used for matching metric names. | +| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | +| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | +| `sort_by` | alpha / cardinality / score | No | | Request how matching metrics should be sorted in the response. | +| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | +| `include_cardinality` | bool | No | false | Request metric cardinality. | +| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | +| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | +| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | +| `next_token` | string | No | | Request the next page of results. | + +**Notes:** + ***search*** The given `search` value will be used to match metric names. @@ -189,9 +196,6 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance * **alpha** - metric names are sorted by alphabetical order. * **cardinality** - metric names are sorted by their cardinality. -* **frequency** - metric names are sorted by their frequency. -* ``` - ``` * **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that `sort_by` is optional, and it is valid for there to be no sorting requested. This allows the server to return / stream results back immediately without the need for any server-side buffering. @@ -257,7 +261,6 @@ Use existing Prometheus API status codes. "results": [ { "name": "activity_tracker_failed_total", - "frequency": 1003, "cardinality": 10, "type": "counter", "help": "How many times has activity tracker failed to insert new activity", @@ -265,7 +268,6 @@ Use existing Prometheus API status codes. }, { "name": "activity_tracker_free_slots", - "frequency": 4, "cardinality": 50, "type": "gauge", "help": "Number of free slots in activity file.", @@ -436,7 +438,7 @@ An endpoint specific to searching for label names. * **alpha** - label names are sorted by alphabetical order. * **cardinality** - label names are sorted by their cardinality. -* **frequency** - label names are sorted by their frequency of use (i.e., the number of active series containing this label within the queried time range). +* **frequency** - label names are sorted by their frequency. * **score** - label names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. #### Response @@ -485,7 +487,7 @@ Use existing Prometheus API status codes. }, { "name": "container", - "frequency": 4, + "frequency": 60, "cardinality": 50 } ] @@ -608,7 +610,7 @@ From an operator perspective, they may not care for the exact value just that it On a similar thought process, a "minimum frequency" could be supported. For instance, the API request could set a `min-frequency=100`. -The server would only return metric/label/value matches if their associated frequency was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. +The server would only return metric/label/value matches if their associated frequency/cardinality was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. With the combination of this min value and the `limit` it would allow the server to return more quickly. @@ -630,8 +632,8 @@ To avoid conflicts, the response record format could be extended to include an ` "frequency": 1003, "extensions": { "mimir": { - "active_series": 10, - "owned_series" : 5, + "some_mimir_specific_attribute": 10, + "another_mimir_specific_attribute" : 5, ... } } @@ -658,7 +660,7 @@ If required for client side applications, a simple endpoint which returns the su ```json { "mimir": { - "active_series": { + "some_mimir_specific_attribute": { "type": "int", "help": "some string explaining the field" }, @@ -683,7 +685,7 @@ These are new endpoints and do not change or alter any existing functionality. N ### Known unknowns * confirm feasibility of including frequency and cardinality in these API responses -* confirm feasibility of supporting cursor based pagination for these new endpoints +* confirm requirement of supporting cursor based pagination for these new endpoints * confirm any performance / response time constraints for these new endpoints * specific choice of fuzzy search algorithm * specific implementation of the search result ordering for auto-complete scenarios @@ -719,6 +721,7 @@ A dedicated endpoint also allows for future enrichments to be added which are on The tasks to do in order to migrate to the new idea. +* [ ] Confirm requirement for supporting pagination or not * [ ] Finalize proposal based on community feedback * [ ] Create Prometheus implementation issue for `/api/v1/search/metric_names` * [ ] Create Prometheus implementation issue for `/api/v1/search/label_names` From 82f6a20ded6ca4df8ba4fcec7b301ec86d20db3d Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 09:48:29 +0800 Subject: [PATCH 21/22] Update default lookback period Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index dfbb5c8..7dd8aad 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -214,7 +214,9 @@ Note that this parameter can only be used with the `limit` parameter. The reques ***start/end*** -It is proposed that these could have default values which align to a reasonable look-back period. ie last 24 hours +It is proposed that these could have default values which align to a reasonable look-back period. + +Ideally this would fall within the WAL. A default look-back of 1 hour is proposed. ***include_*** From d076ecb697d7db17a12d45118dfedf73d7cca83e Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 09:57:58 +0800 Subject: [PATCH 22/22] Add fuzz_alg param Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 108 +++++++++++++----------- 1 file changed, 60 insertions(+), 48 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 7dd8aad..7d19cce 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -148,21 +148,22 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------|----------|---------|------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `search` | string | No | | The search string to be used for matching metric names. | -| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | -| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | -| `sort_by` | alpha / cardinality / score | No | | Request how matching metrics should be sorted in the response. | -| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | -| `include_cardinality` | bool | No | false | Request metric cardinality. | -| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | -| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | -| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | -| `next_token` | string | No | | Request the next page of results. | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------|----------|--------------|------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | string | No | | The search string to be used for matching metric names. | +| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | +| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | +| `sort_by` | alpha / cardinality / score | No | | Request how matching metrics should be sorted in the response. | +| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | +| `include_cardinality` | bool | No | false | Request metric cardinality. | +| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | +| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | +| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | +| `next_token` | string | No | | Request the next page of results. | **Notes:** @@ -192,6 +193,14 @@ A Jaro-Winkler returns a score between 0..1 which can be easily scaled to 0..100 A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100`. mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8. A `fuzz_threshold` of 80 or below would allow this match. +***fuzz_alg*** + +Allow the client to select which fuzzy algorithm is used. Noting that the selection must be supported by the server. + +It is proposed that the default algorithm be Jaro-Winkler, and that all implementations should support this. + +It is proposed that the available fuzzy algorithms be exposed via the `/features` endpoint. + ***sort_by*** * **alpha** - metric names are sorted by alphabetical order. @@ -214,9 +223,9 @@ Note that this parameter can only be used with the `limit` parameter. The reques ***start/end*** -It is proposed that these could have default values which align to a reasonable look-back period. +It is proposed that these could have default values which align to a reasonable look-back period. -Ideally this would fall within the WAL. A default look-back of 1 hour is proposed. +Ideally this would fall within the WAL. A default look-back of 1 hour is proposed. ***include_*** @@ -418,21 +427,22 @@ An endpoint specific to searching for label names. **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|---------|----------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `search` | string | No | | The search string to be used for matching against label names. | -| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | -| `case_sensitive` | bool | No | true | As per above endpoint. | -| `sort_by` | alpha / cardinality / frequency / score | No | | As per above endpoint. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | -| `include_frequency` | bool | No | false | Request label frequency. | -| `include_cardinality` | bool | No | false | Request label cardinality. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `limit` | int >= 0 | No | 100 | As per above endpoint. | -| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `next_token` | string | No | | As per above endpoint. | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|--------------|----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | string | No | | The search string to be used for matching against label names. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / cardinality / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request label frequency. | +| `include_cardinality` | bool | No | false | Request label cardinality. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | +| `next_token` | string | No | | As per above endpoint. | **Notes:** @@ -521,21 +531,22 @@ An endpoint specific to searching for label values. **Query parameters:** -| Name | Type | Required | Default | Description | -|---------------------|---------------------------|----------|---------|-----------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `label` | string | Yes | | The label the user is requesting values for. | -| `search` | string | No | | The search string to be used for matching against label values. | -| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | -| `case_sensitive` | bool | No | true | As per above endpoint. | -| `sort_by` | alpha / frequency / score | No | | As per above endpoint. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | -| `include_frequency` | bool | No | false | Request value frequency. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `limit` | int >= 0 | No | 100 | As per above endpoint. | -| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `next_token` | string | No | | As per above endpoint. | +| Name | Type | Required | Default | Description | +|---------------------|---------------------------|----------|--------------|-----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `label` | string | Yes | | The label the user is requesting values for. | +| `search` | string | No | | The search string to be used for matching against label values. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request value frequency. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | +| `next_token` | string | No | | As per above endpoint. | **Notes:** - The `label` parameter has been deliberately added as a required `query` parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. @@ -655,7 +666,7 @@ The request to enable the extension could be implicit in the requested params. F To be more standards compliant, the extension could be requested in via the `Accept` header. ie `Accept: application/x-ndjson; charset=utf-8; extensions=mimir`. This would also be reflected in the returned `Content-Type`. -If required for client side applications, a simple endpoint which returns the supported extensions and their data model could be exposed. +If required for client side applications, a simple endpoint which returns the supported extensions and their data model could be exposed. Noting that the existing Prometheus `/api/v1/features` could also be used. *GET /api/v1/search/extensions* @@ -728,6 +739,7 @@ The tasks to do in order to migrate to the new idea. * [ ] Create Prometheus implementation issue for `/api/v1/search/metric_names` * [ ] Create Prometheus implementation issue for `/api/v1/search/label_names` * [ ] Create Prometheus implementation issue for `/api/v1/search/label_values` +* [ ] Modify existing Prometheus `/features` endpoint to expose supported fuzzy algorithm choices * [ ] Implement endpoints in Prometheus * [ ] Integrate new endpoints into Prometheus UI * [ ] Update Prometheus documentation