Problem
When downloading GCS backups with multiple parts in parallel, the aggregate download throughput drops significantly at the backup tail — the final phase where only a handful of large parts remain in flight. The root cause is HTTP/2 per-connection flow control.
HTTP/2 multiplexing throughput ceiling
Go's http.Transport negotiates HTTP/2 by default when TLS is used. With HTTP/2, all concurrent part-download streams are multiplexed onto a small number of TCP connections (typically 1-2). Each HTTP/2 connection has a per-connection flow control window that caps the aggregate throughput across all multiplexed streams.
During the main download phase, the per-connection flow control window is shared across many small streams, and the throughput is reasonable. But at the backup tail, only a few large parts remain. These parts compete for the same flow control window, and the window never fully opens — the link is under-utilised.
Why force_http doesn't solve this
The existing gcs.force_http option works around this by forcing HTTP/1.1, but it does so by downgrading the request URL scheme from https:// to http:// via rewriteTransport. This breaks the public GCS endpoint, which returns 403 SSL Required for cleartext requests. force_http is only usable with an internal reverse-proxy cache (e.g., Varnish) that terminates TLS.
Users who need HTTP/1.1 for throughput but connect to GCS directly (or through an HTTPS proxy) cannot use force_http.
Proposed Solution
Add a new GCS config option disable_http2 that suppresses HTTP/2 negotiation (ForceAttemptHTTP2=false, TLSClientConfig.NextProtos=["http/1.1"]) without downgrading the URL scheme. The https:// scheme is preserved, so:
- TLS continues to work against the public GCS endpoint
HTTPS_PROXY (CONNECT-based proxies) continues to work- Each parallel part-download gets its own dedicated TCP connection (up to
MaxIdleConnsPerHost=64)
- Concurrent transfers can saturate the available bandwidth
Configuration
gcs:
disable_http2: true # default: false
Environment variable: GCS_DISABLE_HTTP2
Behavioral difference from force_http
|
force_http |
disable_http2 |
| HTTP version |
HTTP/1.1 |
HTTP/1.1 |
| URL scheme |
http:// (cleartext) |
https:// (TLS preserved) |
| Public GCS endpoint |
403 SSL Required |
Works |
| HTTPS_PROXY |
Broken (scheme is http) |
Works |
| Use case |
Internal cache (Varnish) |
Direct GCS / proxy |
A PR implementing this is forthcoming.
Problem
When downloading GCS backups with multiple parts in parallel, the aggregate download throughput drops significantly at the backup tail — the final phase where only a handful of large parts remain in flight. The root cause is HTTP/2 per-connection flow control.
HTTP/2 multiplexing throughput ceiling
Go's
http.Transportnegotiates HTTP/2 by default when TLS is used. With HTTP/2, all concurrent part-download streams are multiplexed onto a small number of TCP connections (typically 1-2). Each HTTP/2 connection has a per-connection flow control window that caps the aggregate throughput across all multiplexed streams.During the main download phase, the per-connection flow control window is shared across many small streams, and the throughput is reasonable. But at the backup tail, only a few large parts remain. These parts compete for the same flow control window, and the window never fully opens — the link is under-utilised.
Why
force_httpdoesn't solve thisThe existing
gcs.force_httpoption works around this by forcing HTTP/1.1, but it does so by downgrading the request URL scheme fromhttps://tohttp://viarewriteTransport. This breaks the public GCS endpoint, which returns403 SSL Requiredfor cleartext requests.force_httpis only usable with an internal reverse-proxy cache (e.g., Varnish) that terminates TLS.Users who need HTTP/1.1 for throughput but connect to GCS directly (or through an HTTPS proxy) cannot use
force_http.Proposed Solution
Add a new GCS config option
disable_http2that suppresses HTTP/2 negotiation (ForceAttemptHTTP2=false,TLSClientConfig.NextProtos=["http/1.1"]) without downgrading the URL scheme. Thehttps://scheme is preserved, so:HTTPS_PROXY(CONNECT-based proxies) continues to workMaxIdleConnsPerHost=64)Configuration
Environment variable:
GCS_DISABLE_HTTP2Behavioral difference from
force_httpforce_httpdisable_http2http://(cleartext)https://(TLS preserved)A PR implementing this is forthcoming.