Preserving HTTP/1 header case across Envoy accidentally disabled HTTP/2 on a Istio mesh mixing HTTP1 and HTTP2

Preserving HTTP/1 header case across Envoy accidentally disabled HTTP/2 on a Istio mesh mixing HTTP1 and HTTP2#

Motivation#

In a word: we want to support HTTP/2 in an Istio environment running HTTP/1.1 for a long time. So we want a HTTP/1.1 and HTTP2 hybrid mesh.

We want to support HTTP/2 in below flow of APIs between services:

[serviceA app --h2c--> serviceA istio-proxy] ----(http2 over mTLS)---> [serviceB istio-proxy --h2c--> serviceB app]

Environment:

service A: 
  Pod A: 
    ip addr: 192.168.88.94

service B: 10.110.152.25
  Pod B: serviceB-ver-6b54d8c7bc-6vclp
    ip addr: 192.168.33.5

Symptom#

So we try below curl on Pod A:

curl -iv http://serviceB:8080/resource1?p1=v1 \
 -H "Content-Type:application/json" --http2-prior-knowledge
 
*   Trying 10.110.152.25:8080...
* Connected to serviceB (10.110.152.25) port 8080 (#0)
* h2h3 [:method: GET]
* h2h3 [:path: /resource1?p1=v1]
* h2h3 [:scheme: http]
* h2h3 [:authority: serviceB:8080]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* h2h3 [content-type: application/json]
* Using Stream ID: 1 (easy handle 0x557514133e80)
> GET /resource1?p1=v1 HTTP/2
> Host: serviceB:8080
> user-agent: curl/8.0.1
> accept: */*
> content-type:application/json
> 
< HTTP/2 200 
HTTP/2 200 
< content-type: application/json
content-type: application/json
< date: Tue, 07 May 2024 08:44:33 GMT
date: Tue, 07 May 2024 08:44:33 GMT
< x-envoy-upstream-service-time: 19
x-envoy-upstream-service-time: 19
< server: envoy
server: envoy

It seems the app running on Pod A use HTTP/2.

Let us check if Pod B use HTTP/2 :

kubectl logs --tail=1 -f serviceB-ver-6b54d8c7bc-6vclp -c istio-proxy
[2024-05-07T07:18:41.470Z] "GET /resource1?p1=v1 HTTP/1.1" 200 - via_upstream - "-" 0 48 16 14 "-" "curl/8.0.1" "6add007-7242-4983-9862-63cd108e5" "serviceB:8080" "[p8]192.168.88.94[/p8]:8080" outbound|8080|ver|serviceB.ns.svc.cluster.local [p8]192.168.33.5[/p8]:48344 [p8]10.110.152.25[/p8]:8080 [p8]192.168.33.5[/p8]:36650 - -

We can see the istio-proxy of serviceB use HTTP/1.1 protocol.

Glossary#

Background knowledge#

Before the investigation, assuming you have base knowledge of HTTP Meta-data Exchange on ALPN/TLS handshake .

Figure - HTTP protocol meta-data exchange at high level 2

Figure: HTTP protocol meta-data exchange at high level 2#

Open with Draw.io

Investigate#

We know the path of traffic:

[serviceA app --h2c--> serviceA istio-proxy] ----(http2 over mTLS)---> [serviceB istio-proxy --h2c--> serviceB app]

We know serviceA istio-proxy use ALPN to negotiate which version of HTTP used between serviceB istio-proxy. See Better Default Networking – Protocol sniffing

So we run tcpdump on serviceA istio-proxy to inspect ALPN between 2 istio-proxy(s) :

ss -K 'dst 192.168.88.94'

tcpdump -i eth0@if3623 'host 192.168.88.94' -c 1000 -s 65535 -w /tmp/tcpdump.pcap

tshark -r /tmp/tcpdump.pcap -d tcp.port==8080,ssl -2R "ssl" -V | less
...
Transport Layer Security
    TLSv1.3 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 2723
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Extension: application_layer_protocol_negotiation (len=32)
                Type: application_layer_protocol_negotiation (16)
                Length: 32
                ALPN Extension Length: 30
                ALPN Protocol
                    ALPN string length: 14
                    ALPN Next Protocol: istio-http/1.1
                    ALPN string length: 5
                    ALPN Next Protocol: istio
                    ALPN string length: 8
                    ALPN Next Protocol: http/1.1      
...                    

No expected istio-h2 or h2 found.

Debug log of outbound istio-proxy#

Enable debug log of outbound istio-proxy:

curl -XPOST http://localhost:15000/logging\?filter\=trace

Get the log:

{"level":"debug","time":"2024-05-07T07:18:41.471107Z","scope":"envoy filter","msg":"override with 3 ALPNs"}

Evnoy Listener#

So we dump the Envoy configuration of serviceA istio-proxy :

Evnoy Listener on serviceA istio-proxy:

configs:
  dynamic_listeners:
        - name: 0.0.0.0_8080
        active_state:
          version_info: 2024-04-16T09:30:41Z/90
          listener:
            '@type': type.googleapis.com/envoy.config.listener.v3.Listener
            name: 0.0.0.0_8080
            address:
              socket_address:
                address: 0.0.0.0
                port_value: 8080
            filter_chains:
              - filter_chain_match:
                  transport_protocol: raw_buffer
                  application_protocols:
                    - http/1.1
                    - h2c
                filters:
                  - name: envoy.filters.network.http_connection_manager
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                      stat_prefix: outbound_0.0.0.0_8080
                      rds:
...
                      http_filters:
                        - name: envoy.filters.http.grpc_stats
...
                        - name: istio.alpn
                          typed_config:
                            '@type': type.googleapis.com/istio.envoy.config.filter.http.alpn.v2alpha1.FilterConfig
                            alpn_override:
                              - alpn_override:
                                  - istio-http/1.0
                                  - istio
                                  - http/1.0
                              - upstream_protocol: HTTP11
                                alpn_override:
                                  - istio-http/1.1
                                  - istio
                                  - http/1.1
                              - upstream_protocol: HTTP2
                                alpn_override:
                                  - istio-h2
                                  - istio
                                  - h2
...
                        - name: envoy.filters.http.router
                          typed_config:
                            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                      http_protocol_options:
                        header_key_format:
                          stateful_formatter:
                            name: preserve_case
                            typed_config:
                              '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig

If you search istio alpn filter on Google, you may not found any thing meaningful. Only some articles:

So now we know that:

  • When upstream cluster supports HTTP11 , below HTTP protocol will be provided in ALPN of TLS traffic:

                                alpn_override:
                                  - istio-http/1.1
                                  - istio
                                  - http/1.1
  • When upstream cluster supports HTTP2 , below HTTP protocol will be provided in ALPN of TLS traffic:

                              - upstream_protocol: HTTP2
                                alpn_override:
                                  - istio-h2
                                  - istio
                                  - h2

Look back to above tcpdump output, we know that , serviceA istio-proxy assume upstream cluster supported HTTP/1.1 only. Why?

Envoy upstream cluster meta-data declare#

upstream cluster meta-data declare of serviceB on serviceA istio-proxy:

    dynamic_active_clusters:
        cluster:
          '@type': type.googleapis.com/envoy.config.cluster.v3.Cluster
          name: outbound|8080|version|serviceB.ns.svc.cluster.local
          type: EDS
          eds_cluster_config:
            eds_config:
              ads: {}
              initial_fetch_timeout: 0s
              resource_api_version: V3
            service_name: outbound|8080|version|serviceB.ns.svc.cluster.local


          typed_extension_protocol_options:
            envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
              '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
              explicit_http_config:
                http_protocol_options:
                  header_key_format:
                    stateful_formatter:
                      name: preserve_case
                      typed_config:
                        '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig

There are 3 methods of Upstream HTTP protocol selection of Envoy:

  • explicit_http_config : To explicitly configure either HTTP/1 or HTTP/2 (but not both!) use explicit_http_config

  • use_downstream_protocol_config : This allows switching on protocol based on what protocol the downstream connection used.

  • auto_config : This allows switching on protocol based on ALPN. If this is used, the cluster can use either HTTP/1 or HTTP/2, and will use whichever protocol is negotiated by ALPN with the upstream. Clusters configured with AutoHttpConfig will use the highest available protocol; HTTP/2 if supported, otherwise HTTP/1. If the upstream does not support ALPN, AutoHttpConfig will fail over to HTTP/1.

Root cause#

So now we know the direct cause is serviceA istio-proxy assume upstream cluster supported HTTP/1.1 only, and it is cause by above explicit_http_config and it’s sub item http_protocol_options. But why explicit_http_config existed in the upstream cluster meta-data ? It is generated by native Istio ?

Let’s have a look at the EnvoyFilter of Istio:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
  name: mycom-myprd-mesh-preserve-header-case
  namespace: ns
spec:
  configPatches:
  - applyTo: CLUSTER
    patch:
      operation: MERGE
      value:
        typed_extension_protocol_options:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
            explicit_http_config:
              http_protocol_options:
                header_key_format:
                  stateful_formatter:
                    name: preserve_case
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig

  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          http_protocol_options:
            header_key_format:
              stateful_formatter:
                name: preserve_case
                typed_config:
                  '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig

It seems above configuration is copy from HTTP/1.1 Header Casing - from official documentation of Envoy. But developers of Envoy may not think the impaction of explicit_http_config and http_protocol_options when applied on Istio.

There are many github issues on Istio about preserve HTTP/1.1 header case, below list these issues in chronological order:

The conclusion of the discussion is Istio will not support preserve HTTP/1.1 header casing officially:

Issue: Enable preserve HTTP Header casing #32008

We do not intend to ever merge this feature into Istio, as we have medium term plans to use HTTP2 ~everywhere and any http2 hop destroys casing. You can apply EnvoyFilter at your own risk, with the knowledge that it will break sooner or later

So we have to support preserve header casing by Istio Envoy Filter. But for a HTTP/1.1 and HTTP2 hybrid mesh, if you follow HTTP/1.1 Header Casing - from official documentation of Envoy , and use explicit_http_config, you may end up accidentally disabled HTTP/2. So generally speaking use_downstream_protocol_config is a more compatibility and safer choice.

So we can fix it now:

kubectl apply -f - <<"EOF"
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: mycom-myprd-mesh-preserve-header-case
  namespace: ns
spec:
  configPatches:
  - applyTo: CLUSTER
    patch:
      operation: MERGE
      value:
        typed_extension_protocol_options:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
            use_downstream_protocol_config:
              http_protocol_options:
                header_key_format:
                  stateful_formatter:
                    name: preserve_case
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
              http2_protocol_options:
                max_concurrent_streams: 2147483647

  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          http_protocol_options:
            header_key_format:
              stateful_formatter:
                name: preserve_case
                typed_config:
                  '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig

We use use_downstream_protocol_config here because we want the upstream protocol follow the downstream protocol.

Below figure deep dive into the related source code of Envoy Proxy and Istio Proxy. It show you why under the hood.

Figure - upstream http protocol selection troubleshooting

Figure: upstream http protocol selection troubleshooting#

Open with Draw.io

Summary#

Read the official Istio Envoy Filter documentation:

Envoy Filter

EnvoyFilter provides a mechanism to customize the Envoy configuration generated by Istio Pilot. Use EnvoyFilter to modify values for certain fields, add specific filters, or even add entirely new listeners, clusters, etc. This feature must be used with care, as incorrect configurations could potentially destabilize the entire mesh.

May be we should check all configuration items of Istio Envoy Filters by the documentation of Envoy before we apply it to Istio.