Listener connection establishment details#
The process and relationship between event-driven and connection establishment:
The Envoy worker thread hangs in the
epoll_wait()
method. The thread is moved out of the kernel’s runnable queue. the thread sleeps.client establishes a connection. server kernel completes 3 handshakes, triggering a listen socket event.
The operating system moves the Envoy worker thread into the kernel’s runnable queue. the Envoy worker thread wakes up and becomes runnable. the operating system discovers the available cpu resources and schedules the runnable Envoy worker thread onto the cpu (note that runnable and scheduling onto the cpu are not done at the same time). (Note that runnable and scheduling on cpu are not done at the same time)
Envoy analyzes the event list and schedules to different callback functions of FileEventImpl class according to the fd of the event list (see
FileEventImpl::assignEvents
for implementation).The callback function of FileEventImpl class calls the actual business callback function, performs syscall
accept
and completes the socket connection. Get the FD of the new socket:$new_socket_fd
.The business callback function adds
$new_socket_fd
to the epoll listener by callingepoll_ctl
.Return to step 1.
TCP Connection Establishment Procedure#
Take a look at the code to get a general idea of how the connection is established and how it is implemented:
The steps are:
epoll receives the connection request and completes 3 handshakes. It is better to callback to TcpListenerImpl::onSocketEvent().
eventually syscall
accept()
to get the FD of the new socket.call ActiveTcpListener::onAccept()
create a new connection-specific
ListenerFilterChain
.create an
ActiveTcpSocket
dedicated to the new connection and initiate theListenerFilterChain
processExecute the
ListenerFilterChain
process:e.g., TlsInspector::Filter registers to listen for events on the new socket, so that it can read the socket and extract the TLS SNI/ALPN when subsequent events occur on the new socket. 2.
When all
ListenerFilter
s in theListenerFilterChain
have completed all their data exchange and extraction tasks in the new event and event cycle, control of this fd is handed over to one session.
call the core function
ActiveTcpListener::newConnection()
call findFilterChain() to find the best matching
network filter chain configuration
based on the data extracted by theListenerFilter
and the match conditions of eachnetwork filter chain configuration
.Create the
ServerConnection
(a subclass of ConnectionImpl) object.Register the socket event callback to
Network::ConnectionImpl::onFileEvent(uint32_t events)
. This means that future socket events will be handled by thisServerConnection
.
Create a
transportSocket
with thenetwork filter chain configuration
object found earlier.Create a runtime
NetworkFilterChain
with the previously foundnetwork filter chain configuration
object.
Proofing#
If you’re interested in looking at the implementation details, I recommend checking out the articles on my Blog: