Istio Ports and Components#
Each component of Istio listens to a bunch of ports. For beginners, it can be difficult to figure out what each port does. Here, the Figure: Istio Ports and Components illustrates the ports and related functionality of each component when Istio is deployed by default.
Figure: Istio Ports and Components#
The above image needs to be clarified:
The istio-proxy container shares the same Linux
network namespaceas other containers in the same pod.A
network namespaceis a technique used within the kernel to isolate multiple different network configurations. One of these configurations isnetfilter, often referred to asiptables, which we’ll talk about later.
Listening on ports#
You can see which ports you are listening on in the following way:
$ nsenter -n -t $PID_OF_ENVOY
$ ss -ln
u_str LISTEN etc/istio/proxy/SDS 34782 * 0 users:(("pilot-agent",pid=3406,fd=13))
u_str LISTEN etc/istio/proxy/XDS 34783 * 0 users:(("pilot-agent",pid=3406,fd=16))
u_str ESTAB etc/istio/proxy/XDS 1379729 * 1379728 users:(("pilot-agent",pid=3406,fd=8))
u_str ESTAB * 1379728 * 1379729 users:(("envoy",pid=3555,fd=37))
u_str ESTAB etc/istio/proxy/SDS 45274 * 46319 users:(("pilot-agent",pid=3406,fd=15))
u_str ESTAB * 46319 * 45274 users:(("envoy",pid=3555,fd=19))
tcp LISTEN 0.0.0.0:15021 0.0.0.0:* users:(("envoy",pid=3555,fd=40),("envoy",pid=3555,fd=34),("envoy",pid=3555,fd=22))
tcp LISTEN 0.0.0.0:15090 0.0.0.0:* users:(("envoy",pid=3555,fd=39),("envoy",pid=3555,fd=33),("envoy",pid=3555,fd=21))
tcp LISTEN 127.0.0.1:15000 0.0.0.0:* users:(("envoy",pid=3555,fd=18))
tcp LISTEN 0.0.0.0:15001 0.0.0.0:* users:(("envoy",pid=3555,fd=41),("envoy",pid=3555,fd=35),("envoy",pid=3555,fd=31))
tcp LISTEN 127.0.0.1:15004 0.0.0.0:* users:(("pilot-agent",pid=3406,fd=17))
tcp LISTEN 0.0.0.0:15006 0.0.0.0:* users:(("envoy",pid=3555,fd=42),("envoy",pid=3555,fd=36),("envoy",pid=3555,fd=32))
tcp ESTAB 172.21.206.227:40560 10.108.217.90:15012 users:(("pilot-agent",pid=3406,fd=19))
tcp ESTAB 172.21.206.227:43240 10.108.217.90:15012 users:(("pilot-agent",pid=3406,fd=14))
tcp LISTEN *:15020 *:* users:(("pilot-agent",pid=3406,fd=12))
tcp ESTAB 127.0.0.1:35256 127.0.0.1:15020 users:(("envoy",pid=3555,fd=43))
tcp ESTAB 127.0.0.1:35238 127.0.0.1:15020 users:(("envoy",pid=3555,fd=20))
tcp ESTAB [::ffff:127.0.0.1]:15020 [::ffff:127.0.0.1]:35238 users:(("pilot-agent",pid=3406,fd=6))
tcp ESTAB [::ffff:127.0.0.1]:15020 [::ffff:127.0.0.1]:35256 users:(("pilot-agent",pid=3406,fd=18))
iptables#
$ iptables-save
# Generated by iptables-save v1.8.7 on Fri Dec 16 16:18:31 2022
*nat
:PREROUTING ACCEPT [104490:5433496]
:INPUT ACCEPT [105123:5471476]
:OUTPUT ACCEPT [24745:1633905]
:POSTROUTING ACCEPT [42627:2706825]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 201507 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 201507 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 201507 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 201507 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 201507 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 201507 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
Connections#
Figure: Istio Ports and Components contains some descriptions of the TCP connections inside a pod running in Istio (see the ss command output therein). I will not go through them one by one.
Miscellaneous Ops guide#
Packet captures#
Sidecar packet capture#
Figure: Istio Ports and Components contains some instructions for running tcpdumps in Istio. I won’t repeat them here. One thing to highlight is the tcpdump capture point. This point affects the filtering conditions and output of tcpdump. It is importance to know if the capture point is before or after redirect rule of iptables.
tcpdump capture pinpoint:
tcpdump will see inbound traffic before iptables, but will see outbound traffic only after the firewall has processed it.
As a matter of fact, tcpdump is the first software found after the wire (and the NIC, if you will) on the way IN, and the last one on the way OUT.
Wire -> NIC -> tcpdump -> netfilter/iptables
iptables -> tcpdump -> NIC -> Wire
######## sidecar ########
sudo nsenter -t $PID_OF_SIDECAR_ENVOY -n -u
export ETH0_IP=$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
export LOCAL_IP=$(ip addr show lo | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
# inbound mTLS
sudo tcpdump -i eth0 -n -vvvv "(dst port 8080 and dst $ETH0_IP) or (src port 8080 and src $ETH0_IP)"
# inbound plain text
sudo tcpdump -i lo -n -vvvv -A "(dst port 8080 and dst $ETH0_IP) or (src port 8080 and src $ETH0_IP)"
# outbound plain text
sudo tcpdump -i lo -n -vvvv -A "((dst port 15001 and dst 127.0.0.1) or (dst portrange 20000-65535 and dst $ETH0_IP))"
# outbound mTLS
sudo tcpdump -i eth0 -n -vvvv -A "((src portrange 20000-65535 and src $ETH0_IP) or (dst portrange 20000-65535 and dst $ETH0_IP))"
One problem is that outbound plaintext packet capture, outbound IP packet captures 127.0.0.1 after redirect, inbound IP packet captures the ip address before redirect. If you analyze these packets with tools like Wireshark, you will not be able to Follow TCP Stream.
Istio Gateway packet capture#
Generally, Istio Gateway’s upstream (inside the Cluster) and downstream (outside the Cluster) will be in different subnet, so you can use CIDR to distinguish them.
First, let’s look at the CIDR range of a pod in a Kubernetes cluster:
ps -ef | grep cidr
root 48587 20177 0 Dec08 ? 00:21:25 kube-controller-manager ... --cluster-cidr=192.168.0.0/12 ...--service-cluster-ip-range=10.96.0.0/12 ...
Then, attempting to use the above parameter directly will result in an error:
$ sudo tcpdump -i br0 -vvvv -A net 192.168.0.0/12 #168 here
tcpdump: non-network bits set in "192.168.0.0/12"
I found that tcpdump is quite strict on the format of cidr, requiring the first available ip address segment. Using https://cidr.xyz/, we analyzed the first available ip address of 192.168.0.0/12 to be 192.160.0.1, so:
$ sudo tcpdump -i br0 -vvvv -A net 192.160.0.0/12 #160(NOT 168) here
Miraculous 127.0.0.6#
For very many reasons, the ip address that Envoy binds the TCP connection it establishes when inbound traffic goes from Envoy to app is one that ip addr command can’t even find: 127.0.0.6 on the lo interface. If you’re curious, see:
Ending words#
It is very unintuitive and easy to solve network problems under the complex implementation of Istio with the traditional Linux network operation and maintenance approach. While Service Mesh is being applied, observability tools and methods should be changed accordingly. Otherwise, after large-scale use, problem solving will cost quite a lot.