Add support for customizing HTTP headers by kyokuping · Pull Request #1196 · apple/pkl

kyokuping · 2025-08-31T16:28:45Z

This PR adds support for custom HTTP headers, introducing a --http-header CLI flag to accept key=value pairs. These headers can also be specified within the setting.pkl file.

Closes #633

SPICE: apple/pkl-evolution#24

HT154

Thanks for the contribution! 🎉

I have a few primary concerns here:

Although rare, it is valid to send a specific header more than once in a request. When there are headers with the same name their order may also matter.
How does your implementation handle "reserved" headers that users should not override like Origin or User-Agent?
It seems less than ideal to send the same headers to all hosts, eg. if host A requires auth but host B does not, Pkl should not send a configured Authorization header to both hosts. I think a prefix matching scheme like the one used for rewrites is necessary here to ensure the proper security controls are in place.

To this end, I think the Pkl API for this needs to be something along these lines:

headers: Mapping<HttpRewrite, Listing<Pair<String, String>>>

This complicates the CLI a bit, of course, but maybe something like this is reasonable?

--http-header <prefix>='<header name>: <header value>'

HT154 · 2025-09-17T15:58:21Z

Just a heads up, we want reviews from the entire core team because this feature affects the core language and has security implications. One member of the team is currently out on leave so it's going to be a little bit before we can fully review this, but we haven't forgotten about it!

HT154 · 2025-09-17T19:59:58Z

+              val headerParts = header.split(":", limit = 2)
+              require(headerParts.size == 2) { "Header '$header' is not in 'name:value' format. " }
+              PPair(headerParts[0], headerParts[1])


curl accepts this same syntax, but also accepts <name>:[ \t]+<value> and appears to strip the leading (but not trailing) whitespace from header values before sending the request. To avoid surprise for users of Pkl, it might be good to do the same. In theory, HTTP servers should be ignoring this leading (and trailing) whitespace when parsing headers so it shouldn't matter, but it might be best to avoid taking chances here.

I'm also wondering if it makes sense to do some validation of characters in headers here (and in the EvaluatorSettings API) to eagerly report invalid characters instead of lazily doing so when a matching request is made for the first time.

stackoverflow

The design sounds sensible at first glance, but due to the complexity and security issues it may cause it may be a good idea to create a SPICE for it.
What do you think @bioball, @HT154?

bioball

Hey, thanks for the PR! I agree with @stackoverflow; it'd be good to have this discussed in a SPICE.

If you'd like to drive this feature, can you write one up? It should be submitted as PR to pkl-evolution.

Your SPICE should start by copy/pasting the template. Your SPICE number would be the next number after the latest proposed SPICE see (https://github.com/apple/pkl-evolution/pulls)

kyokuping · 2025-09-25T04:29:10Z

I will resume working on this PR after getting the SPICE merged!

stackoverflow · 2025-09-25T08:01:06Z

As a heads up: the SPICE can only be merged once this pr is merged and released on a new version. So you only need the SPICE to be approved.

bioball · 2025-11-13T16:37:28Z

@kyokuping was the close intentional?

kyokuping · 2025-11-13T17:35:48Z

Sorry, accidentally closed the PR. 😢

bioball · 2025-11-13T18:22:33Z

Gotcha, feel free to re-open a new PR!

discount-elf · 2026-01-13T21:48:10Z

Happy to see this, just went looking for http(s) auth support :D

danievanzyl · 2026-03-12T07:50:06Z

When will this be merged? Really looking forward to access private repos 🙏🏻

HT154

Hey @kyokuping would you mind rebasing this to resolve the conflict(s)?

I have some pretty minor feedback, but from a technical standpoint this looks pretty good. I'd like to try and ship this in 0.32 in a few months, so it'd be great if this was ready for another set of eyes by the time the core team's all back together in a few weeks!

HT154 · 2026-03-16T21:01:21Z

+    if (!headerName.equals(headerName.toLowerCase())) {
+      throw new IllegalArgumentException(
+          "HTTP header '%s' should be all lowercase".formatted(headerName));
+    }


Is this validation actually necessary? Why would this be preferable to normalizing header names during CLI parsing?

Based on the previous review on the SPICE, I updated the settings to enforce lowercase values and therefore expected the CLI to match this behavior. If we want to change this in the CLI, we should probably align the EvaluatorSettings implementation too. Which way do you prefer to go with?

In the evaluator settings, "pre"-normalization is required to ensure the Mapping has unique keys. On the CLI, each instance of the option is parsed linearly, so the CLI could perform its own normalization without issue.

In general, I think it's best that we accept the broadest input possible for each input method (even if those are not consistent). It's very possible the other core folks could disagree, so I'm happy to put a pin in this until they're back over the next few weeks.

Hey, had any chance discussing around this?

We just chatted on this; we've settled on: we don't think it's necessary to require any special rules for header names when configured using CLI flags.

Co-authored-by: Jen Basch <jbasch94@gmail.com>

- Introduce pattern-based URL matches - Both accept a single and multiple values for header values - Move header append logic to RequestRewritingClient

Co-authored-by: Jen Basch <jbasch94@gmail.com>

stackoverflow · 2026-05-04T15:31:27Z

+          for ((stringPattern, header) in it) {
+            val headerRegex = Regex("""^(.+?):[ \t]*(.+)$""")


Suggested change

for ((stringPattern, header) in it) {

val headerRegex = Regex("""^(.+?):[ \t]*(.+)$""")

val headerRegex = Regex("""^(.+?):[ \t]*(.+)$""")

for ((stringPattern, header) in it) {

Wasteful to recompile the Regex on every iteration. Even better to move it to the companion object.

stackoverflow · 2026-05-05T13:16:27Z

      }
    }
+
+    val HEADER_NAME_REGEX = Pattern.compile("^[a-zA-Z0-9!#$%&'*+-.^_`|~]+$")


This is duplicated verbatim from IoUtils. You can make it public there and use it here.

stackoverflow · 2026-05-05T13:17:22Z

    }
+
+    val HEADER_NAME_REGEX = Pattern.compile("^[a-zA-Z0-9!#$%&'*+-.^_`|~]+$")
+    val HEADER_VALUE_REGEX = Pattern.compile("^[\\t\\u0020-\\u007E\\u0080-\\u00FF]*$")


Same here (duplicated from IoUtils).

stackoverflow · 2026-05-05T13:52:43Z

      .multiple()
      .toMap()

+  val httpHeaders: List<PPair<Pattern, List<PPair<String, String>>>> by


This is more of a nit, but: List<PPair<Pattern, List<PPair<String, String>>>> is a quite generic type that doesn't convey much information. It would be better to have a properly-named record that stores the URL glob and the header name-value pairs.

I agree this type's gotten a bit out of hand and a record would be a welcome improvement. I'm not eager to block this PR more than necessary, so I'm fine leaving this as follow-up work.

stackoverflow · 2026-05-05T13:54:52Z

+    "via"
+  };
+
+  private static final String[] reservedHeaderPrefixs = {"proxy-", "sec-", "access-control-"};


Suggested change

private static final String[] reservedHeaderPrefixs = {"proxy-", "sec-", "access-control-"};

private static final String[] reservedHeaderPrefixes = {"proxy-", "sec-", "access-control-"};

stackoverflow · 2026-05-05T13:55:31Z

+    "access-control-request-headers",
+    "access-control-request-method",


Do you need these here if you already have "access-control-" in the reservedHeaderPrefixs?

Co-authored-by: Islon Scherer <islonscherer@gmail.com>

HT154

I think this is getting real close now. From my perspective, just a little bit to hash out with URI patterns.

HT154 · 2026-05-12T19:58:11Z

 typealias HttpRewrite = String(startsWith(Regex("https?://")), endsWith("/"), hasNonEmptyHostname)

+@Since { version = "0.32.0" }
+typealias UrlPattern = String(endsWith(Regex("[/*]")))


I think this validation needs to be tightened:

Suggested change

typealias UrlPattern = String(endsWith(Regex("[/*]")))

typealias UrlPattern = String(endsWith(Regex(#"/\*{0,2}"#)))

The idea here is that attempts to match a prefix should always end on a path boundary (exact or prefix) to avoid accidentally matching unintended origins, e.g. pattern *.example.com matches foo.example.company.com, which could leak credentials.

I actually think we don't need to tighten this. Unlike string prefix matching, glob expressions are unlikely to be accidentally misconfigured.

These two are clear enough, in my opinion:

https://example.com** -- match everything that starts with example.com
https://example.com/** -- match everything within the host example.com

I do wonder if we want a external boolean isGlobPattern on class String, so this can be:

String(isGlobPattern)

Hmm I still think it would be ideal to prevent prefix matching on hosts, but I do see what you mean. I like the idea of isGlobPattern to mirror isRegex.

HT154 · 2026-05-12T19:59:19Z

+            IoUtils.validateHeaderName(headerName)
+            IoUtils.validateHeaderValue(headerValue)
+            headersMap
+              .computeIfAbsent(stringPattern) { mutableListOf() }


The pattern here isn't validated by the same rule as the pattern values in pkl:EvaluatorSettings

HT154 · 2026-05-12T20:00:51Z

      .multiple()
      .toMap()

+  val httpHeaders: List<PPair<Pattern, List<PPair<String, String>>>> by


I agree this type's gotten a bit out of hand and a record would be a welcome improvement. I'm not eager to block this PR more than necessary, so I'm fine leaving this as follow-up work.

bioball · 2026-05-12T04:40:44Z

+                      .toList();
+              parsedHeaderDefs.add(new Pair(urlPattern, pairs));
+            } catch (InvalidGlobPatternException e) {
+              throw new PklException(ErrorMessages.create("invalidUri", stringPattern));


Suggested change

throw new PklException(ErrorMessages.create("invalidUri", stringPattern));

throw new PklException(ErrorMessages.create("invalidGlobPattern", stringPattern));

bioball · 2026-05-12T15:50:06Z

  }

+  @Override
+  public Builder setHeaders(List<Pair<Pattern, List<Pair<String, String>>>> headers) {


I think it would be better if this signature matched the in-language evaluator settings.

I think this should be something like:

setHeaderRules(Map<String, Map<String, List<String>>> headerRules)

Where the top-level map key is the glob pattern.
This method can throw IllegalArgumentException if a glob pattern is invalid.

And, I think we should also have:

addHeaderRule(String globPattern, Map<String, List<String>> headers)

It'd also be good to avoid using Pair here.

bioball · 2026-05-12T20:19:42Z

 typealias HttpRewrite = String(startsWith(Regex("https?://")), endsWith("/"), hasNonEmptyHostname)

+@Since { version = "0.32.0" }
+typealias UrlPattern = String(endsWith(Regex("[/*]")))


I actually think we don't need to tighten this. Unlike string prefix matching, glob expressions are unlikely to be accidentally misconfigured.

These two are clear enough, in my opinion:

https://example.com** -- match everything that starts with example.com
https://example.com/** -- match everything within the host example.com

I do wonder if we want a external boolean isGlobPattern on class String, so this can be:

String(isGlobPattern)

bioball

This is in a good enough place to merge; there's some things that I plan on addressing in a follow-up PR!

HT154

With @bioball on this one.

Thanks again for all the work on this, @kyokuping! We really appreciate the contribution.

HT154 reviewed Aug 31, 2025

View reviewed changes

kyokuping requested a review from HT154 September 6, 2025 14:45

HT154 reviewed Sep 17, 2025

View reviewed changes

kyokuping force-pushed the feat/headers branch from 70161b3 to c2632cc Compare September 19, 2025 06:38

stackoverflow reviewed Sep 22, 2025

View reviewed changes

bioball reviewed Sep 22, 2025

View reviewed changes

Comment thread pkl-core/src/main/java/org/pkl/core/http/JdkHttpClient.java Outdated

kyokuping mentioned this pull request Sep 23, 2025

SPICE-0022: Custom HTTP Headers apple/pkl-evolution#24

Open

kyokuping marked this pull request as draft September 25, 2025 04:30

kyokuping closed this Nov 13, 2025

kyokuping force-pushed the feat/headers branch from 94962c3 to ecf2d8b Compare November 13, 2025 15:57

kyokuping reopened this Nov 14, 2025

kyokuping force-pushed the feat/headers branch from e8ea9f6 to ffbf0cf Compare January 17, 2026 14:25

kyokuping force-pushed the feat/headers branch from ffbf0cf to be62d56 Compare March 3, 2026 09:02

kyokuping marked this pull request as ready for review March 3, 2026 13:36

kyokuping requested a review from HT154 March 3, 2026 13:36

HT154 reviewed Mar 12, 2026

View reviewed changes

kyokuping force-pushed the feat/headers branch from 6818324 to f44e39f Compare March 13, 2026 15:20

kyokuping requested a review from HT154 March 16, 2026 15:10

HT154 reviewed Mar 16, 2026

View reviewed changes

kyokuping and others added 3 commits April 24, 2026 01:26

Add support for customizing HTTP headers

5706643

Allow customizing headers for different URLs

e119cab

add @Since annotation in pkl option

5e62b61

Co-authored-by: Jen Basch <jbasch94@gmail.com>

kyokuping and others added 10 commits April 24, 2026 01:26

validate header syntax

d48b858

Rename HttpRewrite to HttpPrefix for clarity

26fff36

Add strict HTTP header validation to EvaluatorSettings

ce945a3

Update implementation based on the SPICE changes

f805e25

- Introduce pattern-based URL matches - Both accept a single and multiple values for header values - Move header append logic to RequestRewritingClient

Revert Copyright date change

0e4078c

Update @Since annotations

857cee0

Rewrite lambdas as expressions

72f1897

Disallow using commas as header separator

487d04c

Implement strict header validation in CLI

39194ba

Use computeIfAbsent

f0c21a0

Co-authored-by: Jen Basch <jbasch94@gmail.com>

kyokuping force-pushed the feat/headers branch from 1e51d66 to f0c21a0 Compare April 23, 2026 16:27

stackoverflow reviewed May 6, 2026

View reviewed changes

kyokuping and others added 7 commits May 8, 2026 17:13

move headerRegex compilation out of loop

97ee16b

remove unused regex

6832d82

fix typos

729673b

update reservedHeaderNames list

c14818c

remove lowercase enforcement for header names in CLI

2433bc3

apply spotless formatting

e4c4754

fix inverted regex condition in IoUtils

5620d56

Co-authored-by: Islon Scherer <islonscherer@gmail.com>

kyokuping requested review from HT154, bioball and stackoverflow May 12, 2026 09:17

HT154 reviewed May 12, 2026

View reviewed changes

bioball reviewed May 12, 2026

View reviewed changes

bioball approved these changes May 12, 2026

View reviewed changes

HT154 approved these changes May 12, 2026

View reviewed changes

bioball merged commit 14085c1 into apple:main May 12, 2026
21 checks passed

		for ((stringPattern, header) in it) {
		val headerRegex = Regex("""^(.+?):[ \t]*(.+)$""")

	private static final String[] reservedHeaderPrefixs = {"proxy-", "sec-", "access-control-"};
	private static final String[] reservedHeaderPrefixes = {"proxy-", "sec-", "access-control-"};

		"access-control-request-headers",
		"access-control-request-method",

	typealias UrlPattern = String(endsWith(Regex("[/*]")))
	typealias UrlPattern = String(endsWith(Regex(#"/\*{0,2}"#)))

	throw new PklException(ErrorMessages.create("invalidUri", stringPattern));
	throw new PklException(ErrorMessages.create("invalidGlobPattern", stringPattern));

Conversation

kyokuping commented Aug 31, 2025 • edited by HT154 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HT154 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HT154 commented Sep 17, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stackoverflow left a comment

Choose a reason for hiding this comment

Uh oh!

bioball left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kyokuping commented Sep 25, 2025

Uh oh!

stackoverflow commented Sep 25, 2025

Uh oh!

bioball commented Nov 13, 2025

Uh oh!

kyokuping commented Nov 13, 2025

Uh oh!

bioball commented Nov 13, 2025

Uh oh!

discount-elf commented Jan 13, 2026

Uh oh!

danievanzyl commented Mar 12, 2026

Uh oh!

HT154 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HT154 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kyokuping commented Aug 31, 2025 •

edited by HT154

Loading

HT154 left a comment •

edited

Loading