Connections being closed -- K8s.Client.Mint.HTTPAdapter Connection closed for reading and writing - stopping this process.

[nickdichev-firework]

Version: 2.5.0

I am having a problem where one of my apps will often (typically ~300-600 times per day) make warning logs that K8s.Client.Mint.HTTPAdapter Connection closed for reading and writing - stopping this process. This is unfortunate because we are triggering sentry on warning logs.

I have seen that there was some recent activity on this issue related to the Mint conn status, however, I am on a more recent version. It seems like an idle timeout, but I am not really sure how to validate. I wanted to experiment with passing some mint options into the adapter ({:keepalive, true}) but I'm not really sure how to do this. Additionally, I guess it could be the server side closing the connection (we are using EKS version 1.27) but I haven't been able to determine what settings the server side is using.

I am also curious why warning log level was chosen here. My understanding is the ConnectionRegistry will reopen the connection, so if one is closed it's not a problem?

Any ideas for things to try? Thanks!

[mruoss]

Hi @nickdichev-firework

Indeed, this is most likely the API server closing the connection after some time. K8s will reconnect. I do admit, the log level is debatable.
Until fixed, I'd like to propose to filter out logs from k8s. It is nice for me to get these logs during development and from users with issues. But I usually don't want logs from libs in my apps.

From https://hexdocs.pm/k8s/observability.html:

All logs produced by this library should have library: :k8s as metadata which can be used in filters.

In order to filter the k8s logs, configure compile_time_purge_matching as follows in your config.exs

# config.exs
config :logger,
  compile_time_purge_matching: [
    [library: :k8s]
  ]

[nickdichev-firework]

@mruoss Great, thanks for the quick response. I did notice that you added metadata to these logger calls so that was going to be my "nuke it" plan!

Do you mind if I leave this issue open so I can be notified if a PR/release goes out to reduce the log level? Thanks!