Preserving HTTP/1 header case across Envoy accidentally disabled HTTP/2 on a Istio mesh mixing HTTP1 and HTTP2#
Motivation#
In a word: we want to support HTTP/2 in an Istio environment running HTTP/1.1 for a long time. So we want a HTTP/1.1 and HTTP2 hybrid mesh.
We want to support HTTP/2 in below flow of APIs between services:
[serviceA app --h2c--> serviceA istio-proxy] ----(http2 over mTLS)---> [serviceB istio-proxy --h2c--> serviceB app]
Environment:
service A:
Pod A:
ip addr: 192.168.88.94
service B: 10.110.152.25
Pod B: serviceB-ver-6b54d8c7bc-6vclp
ip addr: 192.168.33.5
Symptom#
So we try below curl on Pod A:
curl -iv http://serviceB:8080/resource1?p1=v1 \
-H "Content-Type:application/json" --http2-prior-knowledge
* Trying 10.110.152.25:8080...
* Connected to serviceB (10.110.152.25) port 8080 (#0)
* h2h3 [:method: GET]
* h2h3 [:path: /resource1?p1=v1]
* h2h3 [:scheme: http]
* h2h3 [:authority: serviceB:8080]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* h2h3 [content-type: application/json]
* Using Stream ID: 1 (easy handle 0x557514133e80)
> GET /resource1?p1=v1 HTTP/2
> Host: serviceB:8080
> user-agent: curl/8.0.1
> accept: */*
> content-type:application/json
>
< HTTP/2 200
HTTP/2 200
< content-type: application/json
content-type: application/json
< date: Tue, 07 May 2024 08:44:33 GMT
date: Tue, 07 May 2024 08:44:33 GMT
< x-envoy-upstream-service-time: 19
x-envoy-upstream-service-time: 19
< server: envoy
server: envoy
It seems the app running on Pod A use HTTP/2.
Let us check if Pod B use HTTP/2 :
kubectl logs --tail=1 -f serviceB-ver-6b54d8c7bc-6vclp -c istio-proxy
[2024-05-07T07:18:41.470Z] "GET /resource1?p1=v1 HTTP/1.1" 200 - via_upstream - "-" 0 48 16 14 "-" "curl/8.0.1" "6add007-7242-4983-9862-63cd108e5" "serviceB:8080" "[p8]192.168.88.94[/p8]:8080" outbound|8080|ver|serviceB.ns.svc.cluster.local [p8]192.168.33.5[/p8]:48344 [p8]10.110.152.25[/p8]:8080 [p8]192.168.33.5[/p8]:36650 - -
We can see the istio-proxy of serviceB use HTTP/1.1 protocol.
Glossary#
h2c - HTTP/2 over TCP or HTTP/2 Cleartext
h2 - HTTP/2 over TLS (protocol negotiation via ALPN)
Background knowledge#
Before the investigation, assuming you have base knowledge of HTTP Meta-data Exchange on ALPN/TLS handshake .
Investigate#
We know the path of traffic:
[serviceA app --h2c--> serviceA istio-proxy] ----(http2 over mTLS)---> [serviceB istio-proxy --h2c--> serviceB app]
We know serviceA istio-proxy
use ALPN
to negotiate which version of HTTP used between serviceB istio-proxy
. See Better Default Networking – Protocol sniffing
So we run tcpdump on serviceA istio-proxy
to inspect ALPN between 2 istio-proxy(s) :
ss -K 'dst 192.168.88.94'
tcpdump -i eth0@if3623 'host 192.168.88.94' -c 1000 -s 65535 -w /tmp/tcpdump.pcap
tshark -r /tmp/tcpdump.pcap -d tcp.port==8080,ssl -2R "ssl" -V | less
...
Transport Layer Security
TLSv1.3 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 2723
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Extension: application_layer_protocol_negotiation (len=32)
Type: application_layer_protocol_negotiation (16)
Length: 32
ALPN Extension Length: 30
ALPN Protocol
ALPN string length: 14
ALPN Next Protocol: istio-http/1.1
ALPN string length: 5
ALPN Next Protocol: istio
ALPN string length: 8
ALPN Next Protocol: http/1.1
...
No expected istio-h2
or h2
found.
Debug log of outbound istio-proxy#
Enable debug log of outbound istio-proxy:
curl -XPOST http://localhost:15000/logging\?filter\=trace
Get the log:
{"level":"debug","time":"2024-05-07T07:18:41.471107Z","scope":"envoy filter","msg":"override with 3 ALPNs"}
Evnoy Listener#
So we dump the Envoy configuration of serviceA istio-proxy
:
Evnoy Listener on serviceA istio-proxy
:
configs:
dynamic_listeners:
- name: 0.0.0.0_8080
active_state:
version_info: 2024-04-16T09:30:41Z/90
listener:
'@type': type.googleapis.com/envoy.config.listener.v3.Listener
name: 0.0.0.0_8080
address:
socket_address:
address: 0.0.0.0
port_value: 8080
filter_chains:
- filter_chain_match:
transport_protocol: raw_buffer
application_protocols:
- http/1.1
- h2c
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: outbound_0.0.0.0_8080
rds:
...
http_filters:
- name: envoy.filters.http.grpc_stats
...
- name: istio.alpn
typed_config:
'@type': type.googleapis.com/istio.envoy.config.filter.http.alpn.v2alpha1.FilterConfig
alpn_override:
- alpn_override:
- istio-http/1.0
- istio
- http/1.0
- upstream_protocol: HTTP11
alpn_override:
- istio-http/1.1
- istio
- http/1.1
- upstream_protocol: HTTP2
alpn_override:
- istio-h2
- istio
- h2
...
- name: envoy.filters.http.router
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
http_protocol_options:
header_key_format:
stateful_formatter:
name: preserve_case
typed_config:
'@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
If you search istio alpn filter
on Google, you may not found any thing meaningful. Only some articles:
So now we know that:
When upstream cluster supports
HTTP11
, below HTTP protocol will be provided in ALPN of TLS traffic:
alpn_override:
- istio-http/1.1
- istio
- http/1.1
When upstream cluster supports
HTTP2
, below HTTP protocol will be provided in ALPN of TLS traffic:
- upstream_protocol: HTTP2
alpn_override:
- istio-h2
- istio
- h2
Look back to above tcpdump output, we know that , serviceA istio-proxy
assume upstream cluster supported HTTP/1.1
only. Why?
Envoy upstream cluster meta-data declare#
upstream cluster meta-data declare of serviceB
on serviceA istio-proxy
:
dynamic_active_clusters:
cluster:
'@type': type.googleapis.com/envoy.config.cluster.v3.Cluster
name: outbound|8080|version|serviceB.ns.svc.cluster.local
type: EDS
eds_cluster_config:
eds_config:
ads: {}
initial_fetch_timeout: 0s
resource_api_version: V3
service_name: outbound|8080|version|serviceB.ns.svc.cluster.local
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
'@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http_protocol_options:
header_key_format:
stateful_formatter:
name: preserve_case
typed_config:
'@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
There are 3 methods of Upstream HTTP protocol selection of Envoy:
explicit_http_config : To explicitly configure either HTTP/1 or HTTP/2 (but not both!) use
explicit_http_config
use_downstream_protocol_config : This allows switching on protocol based on what protocol the downstream connection used.
auto_config : This allows switching on protocol based on ALPN. If this is used, the cluster can use either HTTP/1 or HTTP/2, and will use whichever protocol is negotiated by ALPN with the upstream. Clusters configured with
AutoHttpConfig
will use the highest available protocol; HTTP/2 if supported, otherwise HTTP/1. If the upstream does not support ALPN,AutoHttpConfig
will fail over to HTTP/1.
Root cause#
So now we know the direct cause is serviceA istio-proxy
assume upstream cluster supported HTTP/1.1
only, and it is cause by above explicit_http_config
and it’s sub item http_protocol_options
. But why explicit_http_config
existed in the upstream cluster meta-data ? It is generated by native Istio ?
Let’s have a look at the EnvoyFilter
of Istio:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
labels:
app.kubernetes.io/managed-by: Helm
name: mycom-myprd-mesh-preserve-header-case
namespace: ns
spec:
configPatches:
- applyTo: CLUSTER
patch:
operation: MERGE
value:
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
'@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http_protocol_options:
header_key_format:
stateful_formatter:
name: preserve_case
typed_config:
'@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
- applyTo: NETWORK_FILTER
match:
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
patch:
operation: MERGE
value:
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
http_protocol_options:
header_key_format:
stateful_formatter:
name: preserve_case
typed_config:
'@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
It seems above configuration is copy from HTTP/1.1 Header Casing - from official documentation of Envoy. But developers of Envoy may not think the impaction of explicit_http_config
and http_protocol_options
when applied on Istio.
There are many github issues on Istio about preserve HTTP/1.1 header case, below list these issues in chronological order:
PR: add support for preserving header key case #33030 - Fixes #32008
Istio Technical Oversight Committee Meeting Notes - 2021/06/7
[howardjohn][10 min] Guidance on breaking change - header casing (https://github.com/istio/istio/pull/33030/)
tl;dr - preserve HTTP key header casing (ie FooBar -> FooBar instead of FooBar -> foobar)
My 2c:
For new users, it seems better in all cases to preserve the case. I don’t see a need to allow an API to lowercase it
For existing users, they might have come to rely on the lowercasing. It seems a bit odd, as most apps probably assume title casing if anything, so hopefully when they adopt Istio they fix to being case insensitive instead of assuming lowercase, but I am sure in practice it may break users
PR: Revert “add support for preserving header key case” #33122 - Reverts #33030
The conclusion of the discussion is Istio will not support preserve HTTP/1.1 header casing officially:
Issue: Enable preserve HTTP Header casing #32008
We do not intend to ever merge this feature into Istio, as we have medium term plans to use HTTP2 ~everywhere and any http2 hop destroys casing. You can apply EnvoyFilter at your own risk, with the knowledge that it will break sooner or later
So we have to support preserve header casing by Istio Envoy Filter
. But for a HTTP/1.1 and HTTP2 hybrid mesh, if you follow HTTP/1.1 Header Casing - from official documentation of Envoy , and use explicit_http_config
, you may end up accidentally disabled HTTP/2. So generally speaking use_downstream_protocol_config
is a more compatibility and safer choice.
So we can fix it now:
kubectl apply -f - <<"EOF"
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: mycom-myprd-mesh-preserve-header-case
namespace: ns
spec:
configPatches:
- applyTo: CLUSTER
patch:
operation: MERGE
value:
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
'@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
use_downstream_protocol_config:
http_protocol_options:
header_key_format:
stateful_formatter:
name: preserve_case
typed_config:
'@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
http2_protocol_options:
max_concurrent_streams: 2147483647
- applyTo: NETWORK_FILTER
match:
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
patch:
operation: MERGE
value:
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
http_protocol_options:
header_key_format:
stateful_formatter:
name: preserve_case
typed_config:
'@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
We use use_downstream_protocol_config
here because we want the upstream protocol follow the downstream protocol.
Below figure deep dive into the related source code of Envoy Proxy and Istio Proxy. It show you why under the hood.
Summary#
Read the official Istio Envoy Filter documentation:
EnvoyFilter
provides a mechanism to customize the Envoy configuration generated by Istio Pilot. Use EnvoyFilter to modify values for certain fields, add specific filters, or even add entirely new listeners, clusters, etc. This feature must be used with care, as incorrect configurations could potentially destabilize the entire mesh.
May be we should check all configuration items of Istio Envoy Filters by the documentation of Envoy before we apply it to Istio.