ADH-6782: Add SPNEGO authentication for History Server & SQL Gateway REST APIs

docs/content/docs/deployment/advanced/historyserver.md

@@ -43,6 +43,7 @@ bin/historyserver.sh (start|start-foreground|stop)
 ```
 
 By default, this server binds to `localhost` and listens at port `8082`.
+If you bind the HistoryServer to a network-facing address such as `0.0.0.0`, enable authentication or place it behind an authenticated gateway. Without authentication, any user with network access can inspect archived job metadata, execution plans, configuration values, timestamps, and configured log links.
 
 Currently, you can only run it as a standalone process.
 
@@ -75,6 +76,33 @@ The contained archives are downloaded and cached in the local filesystem. The lo
 
 Check out the configuration page for a [complete list of configuration options]({{< ref "docs/deployment/config" >}}#history-server).
 
+## SPNEGO Authentication
+
+The HistoryServer web UI and REST endpoints can require Kerberos/SPNEGO authentication. This is opt-in; the default `historyserver.web.authentication.type` is `NONE`.
+
+To enable SPNEGO, configure an HTTP service principal and keytab for the HistoryServer process:
+
+```yaml
+historyserver.web.authentication.type: KERBEROS
+historyserver.web.authentication.kerberos.principal: HTTP/_HOST@EXAMPLE.COM
+historyserver.web.authentication.kerberos.keytab: /etc/security/keytabs/flink-historyserver.keytab
+historyserver.web.authentication.signature.secret-file: /etc/flink/historyserver-auth-secret
+```
+
+The principal must start with `HTTP/`. The `_HOST` placeholder is replaced with the local hostname, and `*` can be used to accept all `HTTP/` principals present in the configured keytab. If `historyserver.web.authentication.kerberos.name-rules` is not configured, the default Hadoop auth-to-local rule is used.
+
+When authentication is enabled, requests without a valid cookie or `Authorization: Negotiate` header return `401` with `WWW-Authenticate: Negotiate`. A valid SPNEGO exchange issues a signed `hadoop.auth` cookie and then proceeds to the existing HistoryServer handlers. Invalid SPNEGO tokens return `403`.
+
+Clients with Kerberos credentials can access the HistoryServer with:
+
+```shell
+curl --negotiate -u : http://historyserver.example.com:8082/jobs/overview
+```
+
+If multiple HistoryServer instances serve the same endpoint, configure the same `historyserver.web.authentication.signature.secret` or `historyserver.web.authentication.signature.secret-file` on all instances so that authentication cookies are accepted consistently. If neither option is configured, a random process-local signing secret is generated at startup.
+
+SPNEGO authenticates the caller only. It does not add per-user or per-group authorization, and it does not change JobManager or TaskManager REST authentication.
+
 ## Log Integration
 
 Flink does not provide built-in methods for archiving logs of completed jobs.

docs/content/docs/deployment/config.md

@@ -161,11 +161,14 @@ You can configure checkpointing directly in code within your Flink job or applic
 
 **Web UI**
 
+  - `web.authentication.type`: Controls authentication for the JobManager Web UI and REST endpoint *(NONE by default)*. Set this to `KERBEROS` to require SPNEGO authentication on all `8081` paths.
   - `web.submit.enable`: Enables uploading and starting jobs through the Flink UI *(true by default)*. Please note that even when this is disabled, session clusters still accept jobs through REST requests (HTTP calls). This flag only guards the feature to upload jobs in the UI.
   - `web.cancel.enable`: Enables canceling jobs through the Flink UI *(true by default)*. Please note that even when this is disabled, session clusters still cancel jobs through REST requests (HTTP calls). This flag only guards the feature to cancel jobs in the UI.
   - `web.upload.dir`: The directory where to store uploaded jobs. Only used when `web.submit.enable` is true.
   - `web.exception-history-size`: Sets the size of the exception history that prints the most recent failures that were handled by Flink for a job.
 
+Exposing the JobManager Web UI / REST endpoint to an untrusted network without authentication allows unauthenticated job submission and control operations. Enable `web.authentication.type: KERBEROS` or place the endpoint behind an authenticated proxy before exposing port `8081`.
+
 **Other**
 
   - `io.tmp.dirs`: The directories where Flink puts local data, defaults to the system temp directory (`java.io.tmpdir` property). If a list of directories is configured, Flink will rotate files across the directories.

docs/content/docs/deployment/security/security-kerberos.md

@@ -60,6 +60,7 @@ The following services and connectors are supported for Kerberos authentication:
 - HDFS
 - HBase
 - ZooKeeper
+- JobManager Web UI / REST endpoint (SPNEGO)
 
 Note that it is possible to enable the use of Kerberos independently for each service or connector.
 For example, the user may enable Hadoop security without necessitating the use of Kerberos for ZooKeeper,
@@ -92,6 +93,34 @@ dynamic entries provided by this module.
 This module configures certain process-wide ZooKeeper security-related settings, namely the ZooKeeper service name (default: `zookeeper`)
 and the JAAS login context name (default: `Client`).
 
+### JobManager Web UI SPNEGO Authentication
+
+The JobManager Web UI and REST endpoint are unauthenticated by default. If port `8081` is reachable from an untrusted network, unauthenticated users can submit jobs and perform control operations. To require Kerberos/SPNEGO authentication on all JobManager WebMonitor paths, enable:
+
+```yaml
+web.authentication.type: KERBEROS
+web.authentication.kerberos.principal: HTTP/_HOST@EXAMPLE.COM
+web.authentication.kerberos.keytab: /etc/security/keytabs/flink-jobmanager-http.keytab
+web.authentication.kerberos.name-rules: DEFAULT
+web.authentication.token.validity: 10 h
+web.authentication.cookie.path: /
+web.authentication.signature.secret-file: /etc/flink/jobmanager-web-auth-secret
+```
+
+The principal must start with `HTTP/`. The `_HOST` placeholder is replaced from `rest.address`, then `rest.bind-address`, then the local canonical hostname. Use `*` to accept all `HTTP/` principals present in the configured keytab.
+
+When enabled, requests without a valid authentication cookie or `Authorization: Negotiate` header receive `401` with `WWW-Authenticate: Negotiate`; malformed or invalid SPNEGO tokens receive `403`. A successful SPNEGO exchange issues a signed `hadoop.auth` cookie, so subsequent requests do not need to negotiate Kerberos again until the cookie expires.
+
+Clients with Kerberos credentials can access the endpoint with:
+
+```bash
+curl --negotiate -u : http://jobmanager.example.com:8081/jobs
+```
+
+For multiple JobManager instances behind the same address, configure a shared `web.authentication.signature.secret` or `web.authentication.signature.secret-file`; otherwise each process uses a random local signing secret and cannot verify cookies issued by another process.
+
+SPNEGO authenticates the caller only. It does not add per-user authorization, user or group allowlists, Ranger policy enforcement, Knox integration, session ownership enforcement, or built-in Flink CLI / REST client SPNEGO support.
+
 ## Deployment Modes
 Here is some information specific to each deployment mode.  
 

docs/content/docs/dev/table/sql-gateway/overview.md

@@ -57,6 +57,10 @@ $ ./bin/sql-gateway.sh start -Dsql-gateway.endpoint.rest.address=localhost
 The command starts the SQL Gateway with REST Endpoint that listens on the address localhost:8083. You can use the curl command to check
 whether the REST Endpoint is available.
 
+{{< hint warning >}}
+The SQL Gateway REST endpoint does not authenticate requests by default. If it is bound to a network-facing address, any client with network access can create sessions and execute SQL unless you enable REST SPNEGO authentication or protect the endpoint with external access controls.
+{{< /hint >}}
+
 ```bash
 $ curl http://localhost:8083/v1/info
 {"productName":"Apache Flink","version":"{{< version >}}"}

docs/content/docs/dev/table/sql-gateway/rest.md

@@ -88,9 +88,85 @@ Endpoint Options
             <td>Integer</td>
             <td>The port that the client connects to. If bind-port has not been specified, then the sql gateway server will bind to this port.</td>
         </tr>
+        <tr>
+            <td><h5>sql-gateway.endpoint.rest.authentication.type</h5></td>
+            <td style="word-wrap: break-word;">NONE</td>
+            <td>Enum</td>
+            <td>Authentication type for the SQL Gateway REST endpoint. Supported values are NONE and KERBEROS.</td>
+        </tr>
+        <tr>
+            <td><h5>sql-gateway.endpoint.rest.authentication.kerberos.principal</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Kerberos principal for accepting SPNEGO requests when REST authentication type is KERBEROS. The value supports HTTP/_HOST@REALM host replacement and * to load all HTTP principals from the keytab.</td>
+        </tr>
+        <tr>
+            <td><h5>sql-gateway.endpoint.rest.authentication.kerberos.keytab</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Path to the keytab containing the REST SPNEGO service principal. This option is required when REST authentication type is KERBEROS.</td>
+        </tr>
+        <tr>
+            <td><h5>sql-gateway.endpoint.rest.authentication.kerberos.name-rules</h5></td>
+            <td style="word-wrap: break-word;">DEFAULT</td>
+            <td>String</td>
+            <td>Kerberos auth-to-local rules used to map the authenticated Kerberos principal to a local user name.</td>
+        </tr>
+        <tr>
+            <td><h5>sql-gateway.endpoint.rest.authentication.token.validity</h5></td>
+            <td style="word-wrap: break-word;">10 h</td>
+            <td>Duration</td>
+            <td>Validity of the signed REST authentication cookie issued after a successful SPNEGO exchange.</td>
+        </tr>
+        <tr>
+            <td><h5>sql-gateway.endpoint.rest.authentication.cookie.path</h5></td>
+            <td style="word-wrap: break-word;">/</td>
+            <td>String</td>
+            <td>Path attribute for the REST authentication cookie.</td>
+        </tr>
+        <tr>
+            <td><h5>sql-gateway.endpoint.rest.authentication.signature.secret</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Shared secret used to sign REST authentication cookies. If neither this option nor signature.secret-file is configured, a random process-local secret is generated.</td>
+        </tr>
+        <tr>
+            <td><h5>sql-gateway.endpoint.rest.authentication.signature.secret-file</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Path to a file containing the shared secret used to sign REST authentication cookies. Configure either signature.secret or signature.secret-file, but not both.</td>
+        </tr>
     </tbody>
 </table>
 
+REST SPNEGO Authentication
+----------------
+
+By default, the REST endpoint does not authenticate requests. If the endpoint is reachable from an untrusted network, any client that can connect to it can create sessions, execute SQL, submit jobs, and access configured data sources. Bind the endpoint to a trusted interface, place it behind an authenticated gateway, or enable SPNEGO authentication before exposing it beyond a trusted boundary.
+
+SPNEGO authentication is opt-in and applies to every SQL Gateway REST path, including `/v*/info`.
+
+```yaml
+sql-gateway.endpoint.rest.authentication.type: KERBEROS
+sql-gateway.endpoint.rest.authentication.kerberos.principal: HTTP/_HOST@EXAMPLE.COM
+sql-gateway.endpoint.rest.authentication.kerberos.keytab: /etc/security/keytabs/flink-sql-gateway.keytab
+sql-gateway.endpoint.rest.authentication.kerberos.name-rules: DEFAULT
+sql-gateway.endpoint.rest.authentication.signature.secret-file: /etc/flink/sql-gateway-cookie-secret
+```
+
+Clients must use an HTTP client that can negotiate SPNEGO, for example:
+
+```bash
+$ kinit user@EXAMPLE.COM
+$ curl --negotiate -u : http://sql-gateway-host:8083/v1/info
+$ curl --negotiate -u : --request POST http://sql-gateway-host:8083/v1/sessions
+```
+
+Unauthenticated requests receive `401` with `WWW-Authenticate: Negotiate`. Malformed or invalid SPNEGO tokens receive `403`.
+After a successful SPNEGO exchange, the REST endpoint issues a signed `hadoop.auth` cookie. Multi-instance or load-balanced SQL Gateway deployments must configure the same `sql-gateway.endpoint.rest.authentication.signature.secret` or `sql-gateway.endpoint.rest.authentication.signature.secret-file` on every instance; otherwise cookies issued by one instance will not be accepted by another.
+
+The built-in Flink SQL Client and Flink JDBC Driver do not negotiate SPNEGO with the SQL Gateway REST endpoint in this change. Use an HTTP SPNEGO-capable client or put the SQL Gateway behind a separate authenticated gateway or proxy when those clients are required.
+
 REST API
 ----------------
 

docs/layouts/shortcodes/generated/history_server_configuration.html

@@ -56,6 +56,54 @@
             <td>String</td>
             <td>Address of the HistoryServer's web interface.</td>
         </tr>
+        <tr>
+            <td><h5>historyserver.web.authentication.cookie.path</h5></td>
+            <td style="word-wrap: break-word;">"/"</td>
+            <td>String</td>
+            <td>Cookie path for HistoryServer SPNEGO authentication cookies.</td>
+        </tr>
+        <tr>
+            <td><h5>historyserver.web.authentication.kerberos.keytab</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Absolute path to the keytab file for the HistoryServer SPNEGO principal. Required when HistoryServer web authentication type is KERBEROS.</td>
+        </tr>
+        <tr>
+            <td><h5>historyserver.web.authentication.kerberos.name-rules</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Kerberos auth-to-local rules used to map authenticated principals to local user names. If unset, the default rule is used.</td>
+        </tr>
+        <tr>
+            <td><h5>historyserver.web.authentication.kerberos.principal</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Kerberos principal for HistoryServer SPNEGO authentication. Required when HistoryServer web authentication type is KERBEROS. The principal must start with HTTP/. The _HOST placeholder is replaced with the local hostname. Use * to accept all HTTP principals in the keytab.</td>
+        </tr>
+        <tr>
+            <td><h5>historyserver.web.authentication.signature.secret</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Shared secret for signing HistoryServer SPNEGO authentication cookies. If neither this option nor the secret-file option is set, a random process-local secret is generated. Configure the same secret for all HistoryServer instances that should accept each other's authentication cookies.</td>
+        </tr>
+        <tr>
+            <td><h5>historyserver.web.authentication.signature.secret-file</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>File containing the shared secret for signing HistoryServer SPNEGO authentication cookies. Configure the same secret file for all HistoryServer instances that should accept each other's authentication cookies.</td>
+        </tr>
+        <tr>
+            <td><h5>historyserver.web.authentication.token.validity</h5></td>
+            <td style="word-wrap: break-word;">10 h</td>
+            <td>Duration</td>
+            <td>Validity period for HistoryServer SPNEGO authentication cookies.</td>
+        </tr>
+        <tr>
+            <td><h5>historyserver.web.authentication.type</h5></td>
+            <td style="word-wrap: break-word;">NONE</td>
+            <td><p>Enum</p></td>
+            <td>Authentication type for the HistoryServer web frontend and REST endpoints.<br /><br />Possible values:<ul><li>"NONE"</li><li>"KERBEROS"</li></ul></td>
+        </tr>
         <tr>
             <td><h5>historyserver.web.port</h5></td>
             <td style="word-wrap: break-word;">8082</td>

docs/layouts/shortcodes/generated/sql_gateway_rest_configuration.html

@@ -14,6 +14,54 @@
             <td>String</td>
             <td>The address that should be used by clients to connect to the sql gateway server.</td>
         </tr>
+        <tr>
+            <td><h5>authentication.cookie.path</h5></td>
+            <td style="word-wrap: break-word;">"/"</td>
+            <td>String</td>
+            <td>Cookie path used for SQL Gateway REST authentication cookies.</td>
+        </tr>
+        <tr>
+            <td><h5>authentication.kerberos.keytab</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Kerberos keytab used by the SQL Gateway REST endpoint SPNEGO acceptor.</td>
+        </tr>
+        <tr>
+            <td><h5>authentication.kerberos.name-rules</h5></td>
+            <td style="word-wrap: break-word;">"DEFAULT"</td>
+            <td>String</td>
+            <td>Kerberos auth-to-local rules used to derive the local user name from the authenticated Kerberos principal.</td>
+        </tr>
+        <tr>
+            <td><h5>authentication.kerberos.principal</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Kerberos principal used by the SQL Gateway REST endpoint SPNEGO acceptor. The HTTP/_HOST@REALM pattern is supported, and '*' uses all HTTP principals from the keytab.</td>
+        </tr>
+        <tr>
+            <td><h5>authentication.signature.secret</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Shared secret used to sign SQL Gateway REST authentication cookies. If neither this option nor authentication.signature.secret-file is configured, a random per-process secret is used.</td>
+        </tr>
+        <tr>
+            <td><h5>authentication.signature.secret-file</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>File containing the shared secret used to sign SQL Gateway REST authentication cookies. If neither this option nor authentication.signature.secret is configured, a random per-process secret is used.</td>
+        </tr>
+        <tr>
+            <td><h5>authentication.token.validity</h5></td>
+            <td style="word-wrap: break-word;">10 h</td>
+            <td>Duration</td>
+            <td>Validity period of the SQL Gateway REST authentication cookie.</td>
+        </tr>
+        <tr>
+            <td><h5>authentication.type</h5></td>
+            <td style="word-wrap: break-word;">NONE</td>
+            <td><p>Enum</p></td>
+            <td>The authentication type for the SQL Gateway REST endpoint. Supported values are NONE and KERBEROS.<br /><br />Possible values:<ul><li>"NONE"</li><li>"KERBEROS"</li></ul></td>
+        </tr>
         <tr>
             <td><h5>bind-address</h5></td>
             <td style="word-wrap: break-word;">(none)</td>