I'm not sure but I think the wrapper is not closing closed connections properly:
Details
good
$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.2:34008 TIME_WAIT
tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.3:50373 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55576 unbound.default.svc.:53 TIME_WAIT
tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.2:50374 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55574 unbound.default.svc.:53 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55578 unbound.default.svc.:53 TIME_WAIT
tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.3:50371 TIME_WAIT
tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.2:50367 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55556 unbound.default.svc.:53 TIME_WAIT
tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.3:50372 TIME_WAIT
tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.3:50190 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55558 unbound.default.svc.:53 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55582 unbound.default.svc.:53 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55580 unbound.default.svc.:53 TIME_WAIT
tcp 0 0 dnscrypt-768656ff6d:443 10.56.2.1:20834 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55586 unbound.default.svc.:53 TIME_WAIT
tcp 0 0 dnscrypt-768656ff6d:443 10.56.2.1:20834 TIME_WAIT
tcp 0 0 dnscrypt-768656ff:55586 unbound.default.svc.:53 TIME_WAIT
udp 0 0 0.0.0.0:48047 0.0.0.0:*
udp 0 0 0.0.0.0:443 0.0.0.0:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
As it happens
$ netstat -a -n
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 10.56.2.13:43356 10.59.242.77:53 TIME_WAIT
tcp 0 0 10.56.2.13:443 10.152.0.3:52602 TIME_WAIT
tcp 0 0 10.56.2.13:43328 10.59.242.77:53 TIME_WAIT
tcp 0 0 10.56.2.13:443 10.152.0.2:52641 TIME_WAIT
tcp 323 0 10.56.2.13:443 10.56.2.1:57788 CLOSE_WAIT
tcp 0 0 10.56.2.13:443 10.152.0.3:51751 TIME_WAIT
tcp 0 0 10.56.2.13:43390 10.59.242.77:53 TIME_WAIT
udp 0 0 0.0.0.0:48010 0.0.0.0:*
udp 0 0 0.0.0.0:443 0.0.0.0:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
Bad (no more queries are being answered untill a dnscrypt-wrapper restart)
$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 323 0 dnscrypt-768656ff6d:443 10.152.0.3:3232 CLOSE_WAIT
tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.3:53468 CLOSE_WAIT
tcp 195 0 dnscrypt-768656ff6d:443 10.152.0.2:34023 CLOSE_WAIT
tcp 323 0 dnscrypt-768656ff6d:443 10.152.0.3:1908 CLOSE_WAIT
tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.2:53527 CLOSE_WAIT
tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.3:49746 CLOSE_WAIT
tcp 259 0 dnscrypt-768656ff6d:443 10.152.0.2:58956 CLOSE_WAIT
tcp 259 0 dnscrypt-768656ff6d:443 10.152.0.3:32736 CLOSE_WAIT
tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.2:49736 CLOSE_WAIT
tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.2:20808 CLOSE_WAIT
tcp 195 0 dnscrypt-768656ff6d:443 10.152.0.3:3420 CLOSE_WAIT
tcp 259 0 dnscrypt-768656ff6d:443 10.152.0.3:1915 CLOSE_WAIT
tcp 323 0 dnscrypt-768656ff6d:443 10.152.0.2:58366 CLOSE_WAIT
tcp 195 0 dnscrypt-768656ff6d:443 10.152.0.2:3404 CLOSE_WAIT
tcp 387 0 dnscrypt-768656ff6d:443 10.152.0.3:35672 CLOSE_WAIT
tcp 48 0 dnscrypt-768656ff6d:443 10.56.2.1:9866 CLOSE_WAIT
tcp 387 0 dnscrypt-768656ff6d:443 10.152.0.3:3424 CLOSE_WAIT
tcp 195 0 dnscrypt-768656ff6d:443 10.152.0.2:3416 CLOSE_WAIT
udp 0 0 0.0.0.0:443 0.0.0.0:*
udp 0 0 0.0.0.0:54437 0.0.0.0:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
$ ss -tano
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:443 *:*
CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.3:30367
CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.3:9494
CLOSE-WAIT 195 0 10.56.2.13:443 10.56.2.1:52484
CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.2:56356
CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.2:14286
CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.2:52527
CLOSE-WAIT 48 0 10.56.2.13:443 10.56.2.1:29095
CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.2:4251
CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.3:61126
CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.2:14283
CLOSE-WAIT 131 0 10.56.2.13:443 10.152.0.2:7763
CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.3:52521
CLOSE-WAIT 131 0 10.56.2.13:443 10.152.0.2:14285
CLOSE-WAIT 195 0 10.56.2.13:443 10.56.2.1:52524
CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.2:50186
CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.3:31341
CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.2:7767
CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.3:9773
CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.3:61116
CLOSE-WAIT 323 0 10.56.2.13:443 10.56.2.1:52501
CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.3:14269
CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.3:7758
CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.3:30361
CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.3:49210
CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.2:52517
ESTAB 322 0 10.56.2.13:443 10.152.0.2:52531
CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.2:14268
CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.3:29382
CLOSE-WAIT 323 0 10.56.2.13:443 10.56.2.1:52483
CLOSE-WAIT 259 0 10.56.2.13:443 10.56.2.1:52502
CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.3:52498
CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.2:7764
CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.3:52499
CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.3:16982
I'm using GCP with kubernetes. So traffic routed like this: GCP LoadBalancer->kubernetes-service->dnscrypt-wrapper-container->kubernetes-service->unbound-container
Restarting dnscrypt-wrapper temporarily fixes the problem
I'm not sure but I think the wrapper is not closing closed connections properly:
Details
good
As it happens
Bad (no more queries are being answered untill a dnscrypt-wrapper restart)
I'm using GCP with kubernetes. So traffic routed like this: GCP LoadBalancer->kubernetes-service->dnscrypt-wrapper-container->kubernetes-service->unbound-container
Restarting dnscrypt-wrapper temporarily fixes the problem