ArgumentError in K8s.Client.Mint.ConnectionRegistry on TCP Connection

[Rshep3087]

Description

Encountering an ArgumentError when K8s.Client.Mint.ConnectionRegistry attempts to establish a TCP connection via the Mint library. This issue leads to the termination of the GenServer managing the connections.

Error Message and Stack Trace

11:52:26.810 [error] GenServer K8s.Client.Mint.ConnectionRegistry terminating
** (ArgumentError) argument error
    (kernel 9.0.2) gen_tcp.erl:227: :gen_tcp.connect/4
    (mint 1.5.2) lib/mint/core/transport/tcp.ex:41: Mint.Core.Transport.TCP.connect/3
    (mint 1.5.2) lib/mint/http1.ex:133: Mint.HTTP1.connect/4
    (k8s 2.6.0) lib/k8s/client/mint/connection_registry.ex:107: K8s.Client.Mint.ConnectionRegistry.handle_call/3
    (stdlib 5.0.2) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.0.2) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.0.2) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.98.0>): {:get_or_open, {:http, "localhost", 18450, [transport_opts: [verify: :verify_none, cacertfile: ~c"/Users/ryan/Library/Caches/mix/installs/elixir-1.15.4-erts-14.0.2/be31dcf1afec9e679df0d6bfbc58eed4/_build/dev/lib/castore/priv/cacerts.pem"]]}}
State: {%{}, %{}}
Client #PID<0.98.0> is alive

    (stdlib 5.0.2) gen.erl:240: :gen.do_call/4
    (elixir 1.15.4) lib/gen_server.ex:1071: GenServer.call/3
    (k8s 2.6.0) lib/k8s/client/mint/connection_registry.ex:53: K8s.Client.Mint.ConnectionRegistry.checkout/1
    (k8s 2.6.0) lib/k8s/client/mint_http_provider.ex:80: K8s.Client.MintHTTPProvider.do_stream_to/6
    (k8s 2.6.0) lib/k8s/client/mint_http_provider.ex:34: K8s.Client.MintHTTPProvider.stream/5
    (k8s 2.6.0) lib/k8s/client/mint_http_provider.ex:16: K8s.Client.MintHTTPProvider.request/5
    (k8s 2.6.0) lib/k8s/discovery/driver/http.ex:17: K8s.Discovery.Driver.HTTP.resources/3
    (k8s 2.6.0) lib/k8s/discovery/resource_finder.ex:38: K8s.Discovery.ResourceFinder.find_resource/3

Steps to Reproduce

1. Run as an elixir script

Mix.install([{:k8s, "~> 2.0"}])

{:ok, conn} =
  K8s.Conn.from_file("~/.kube/config",
    insecure_skip_tls_verify: true
  )

operation = K8s.Client.list("apps/v1", "Deployment", namespace: "default")

{:ok, deployments} = K8s.Client.run(conn, operation)

IO.inspect(deployments)

Environment

❯ elixir --version
Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Elixir 1.15.4 (compiled with Erlang/OTP 26)

[mruoss]

TLDR: Remove insecure_skip_tls_verify: true when connecting to this cluster.

Okay, I have never seen this before and your script works like a charm over here.

That being said, I think the problem with your script lies in the combination of the endpoint of your API server (http://localhost and insecure_skip_tls_verify: true). Since you're not connecting via SSL (http vs. https), the TLS instruction should not be passed and probably leats to the argument error you see. Try omitting it for this cluster.

Also, this option insecure_skip_tls_verify is not going to be needed at all anymore, once a new version of mint is out or when using Erlang OTP 27:

• Add match_fun clause to deal with ip addresses in TLS handshake elixir-mint/mint#418
• public_key:pkix_verify_hostname/N returns {:bad_cert, :hostname_check_failed} when connecting to IP addresses erlang/otp#7968

[Rshep3087]

Thanks for the response. Ripped insecure_skip_tls_verify out and still get the same issue. Kubectl, k9s, etc all work using the context that I am using. Tried this in Livebook and still get the same issue.

[mruoss]

Oooh I see. We pass verify: verify_peer by default. But we should only do that if it's actually an SSL endpoint.