Troubleshooting - 问题排查

• Decrypt and Dump TLS Traffic - 解密与监听 TLS 流量
  • TLS key log feature
    • Envoy Key Log 配置
      • extensions.transport_sockets.tls.v3.TlsKeyLog
  • Decryption Tools
  • Key Log Format
  • Ref
• TCP Proxy half-closed connection leak for 1 hour in some scenarios
  • Base knowledge
    • Conntrack base knowledge
    • TCP
    • Conntrack table
  • Environment
    • Testing POD
    • TCP service pod
    • enable conntrack’s invalid packet log on Testing POD
    • skills
      • nsenter network namespace
  • socket leak & occupy on FIN_WAIT2
    • 1. TCPProxy: connections ESTABLISHED
    • 2. TCPProxy: upstream service active close connection
    • 3. after 60s, conntrack table CLOSE_WAIT entry timeout
    • 4. app close connection - FIN not reach peer
• App outbound connecting timed out because App selected a ephemeral port which collisions with the existing socket on 15001(outbound) listener
  • Base knowledge
    • TCP
      • TCP Challenge ACK
    • Conntrack table
  • Environment
    • Testing POD
    • TCP service pod
    • enable conntrack’s invalid packet log on Testing POD
  • New connection timeout
    • App build connection on the same ephemeral port
      • Case 1: connect timed out by collision ephemeral port
      • Case 2: connect by collision ephemeral port but seq-no happens to be in the TCP window of the old connection
  • Skills
• 因配置 Envoy 保留 HTTP/1 header 大小写，在混合 HTTP1 和 HTTP2 的 Istio 网格上意外禁用了 HTTP/2
  • 动机
  • 症状
  • 术语
  • 背景知识
  • 调查
    • outbound istio-proxy 的 debug log
    • Evnoy Listener
    • Envoy upstream cluster 的 meta-data
  • 问题根源
  • 结语