Dave Cottlehuber
2018-11-15 10:14:44 UTC
bonjour list,
In comparison to 1.8 (and probably 1.9b5 but I can't verify that at present) the 301 redirect seems to be handled differently. Initially I thought this was an HTTP2 issue but it 's reproducible on HTTP/1.1 as well.
curl --http1.1 -4vsSLo /dev/null https://logs.example.com/ > h11_18.log or h11_19b6.log)
full diff of logs are at end of the email.
Interestingly this is handled differently in browsers - firefox compiles strictly with the redirect and eventually exceeds its acceptable URL length, appending the %20HTTP/1.1 each time. Chrome and forks seem to ignore it.
the curl output is identical until:
|< HTTP/1.1 301 Moved Permanently
|< Content-length: 0
-|< Location: https://example.com/
+|< Location: https://example.com/ HTTP/1.1
where we can see under 19b6 the HTTP/1.1 sneaks in.
And as we are using the -L follow option in curl we see the incorrect URL being propagated back:
|* Connection #0 to host logs.example.com left intact
-|* Issue another request to this URL: 'https://example.com/'
+|* Issue another request to this URL: 'https://example.com/ HTTP/1.1'
|* Trying 95.216.20.215...
I've tried fiddling with my 301 settings in haproxy.conf to no avail.
environment:
FreeBSD 11.2Rp4 amd64
internet -> haproxy :443 (ipv4 or ipv6) -> h2o 2.3.0b1 backend for serving actual files.
# /usr/local/etc/haproxy/haproxy.conf
... blah
bind ipv4@:443 ssl alpn h2, crt ...
bind ipv6@:443 ssl alpn h2, crt ...
# redirect anything that doesn't match our ACLs or isn't TLS
http-request redirect code 301 location https://example.com%[capture.req.uri] unless www or api or beta
http-request redirect scheme https code 301 if !{ ssl_fc }
I can provide full actual logs / configs off-list if needed, and it's quick to switch in versions of haproxy for validating or get pcap traces. Just bug me on irc dch in #haproxy channel.
haproxy -vvv
HA-Proxy version 1.9-dev6 2018/11/11
*** or HA-Proxy version 1.8.14-52e4d43 2018/09/20
Copyright 2000-2018 Willy Tarreau <***@haproxy.org>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.5
Built with OpenSSL version : OpenSSL 1.0.2o-freebsd 27 Mar 2018
Running on OpenSSL version : OpenSSL 1.0.2o-freebsd 27 Mar 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available multiplexer protocols :
(protocols markes as <default> cannot be specified using 'proto' keyword)
<default> : mode=TCP|HTTP side=FE|BE
h2 : mode=HTTP side=FE
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
patdiff of curl output between http/1.1 and 1.8 vs http/1.1 and 1.9b6:
------ h11_18.log
++++++ h11_19b6.log
@|-39,16 +39,16 ============================================================
|> User-Agent: curl/7.62.0
|> Accept: */*
|>
|{ [5 bytes data]
|< HTTP/1.1 301 Moved Permanently
|< Content-length: 0
!|< Location: https://example.com/ HTTP/1.1
|<
|* Connection #0 to host logs.example.com left intact
-|* Issue another request to this URL: 'https://example.com/'
+|* Issue another request to this URL: 'https://example.com/ HTTP/1.1'
|* Trying 95.216.20.215...
|* TCP_NODELAY set
|* Connected to example.com (95.216.20.215) port 443 (#1)
|* ALPN, offering http/1.1
|* successfully set certificate verify locations:
|* CAfile: /usr/local/share/certs/ca-root-nss.crt
@|-79,36 +79,25 ============================================================
|* start date: Sep 6 19:45:18 2018 GMT
|* expire date: Dec 5 19:45:18 2018 GMT
|* subjectAltName: host "example.com" matched cert's "example.com"
|* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
|* SSL certificate verify ok.
|} [5 bytes data]
!|> GET / HTTP/1.1 HTTP/1.1
|> Host: example.com
|> User-Agent: curl/7.62.0
|> Accept: */*
|>
|{ [5 bytes data]
-|< HTTP/1.1 200 OK
-|< Content-Length: 11297
-|< Server: h2o/2.3.0-beta1
-|< date: Thu, 15 Nov 2018 01:17:58 GMT
-|< content-type: text/html; charset=utf-8
-|< last-modified: Thu, 20 Sep 2018 21:29:23 GMT
-|< etag: "5ba41133-2c21"
-|< vary: accept-encoding
-|< accept-ranges: bytes
-|< content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline' https://cloud.typogra
-|< X-koans: i09
-|< X-sni: example.com
-|< x-powered-by: FreeBSD
-|< x-frame-options: deny
-|< X-XSS-Protection: 1;mode=block
-|< X-Content-Type-Options: nosniff
-|< X-UA-Compatible: IE=Edge
-|< Referrer-Policy: strict-origin-when-cross-origin
-|< Access-Control-Allow-Origin: example.com
-|< Strict-Transport-Security: max-age=15552000
-|< Cache-Control: no-transform
+|* HTTP 1.0, assume close after body
+|< HTTP/1.0 400 Bad request
+|< Cache-Control: no-cache
+|< Connection: close
+|< Content-Type: text/html
|<
-|{ [11297 bytes data]
-|* Connection #1 to host example.com left intact
+|{ [90 bytes data]
+|* TLSv1.2 (IN), TLS alert, close notify (256):
+|{ [2 bytes data]
+|* Closing connection 1
+|} [5 bytes data]
+|* TLSv1.2 (OUT), TLS alert, close notify (256):
+|} [2 bytes data]
thanks
Dave
In comparison to 1.8 (and probably 1.9b5 but I can't verify that at present) the 301 redirect seems to be handled differently. Initially I thought this was an HTTP2 issue but it 's reproducible on HTTP/1.1 as well.
curl --http1.1 -4vsSLo /dev/null https://logs.example.com/ > h11_18.log or h11_19b6.log)
full diff of logs are at end of the email.
Interestingly this is handled differently in browsers - firefox compiles strictly with the redirect and eventually exceeds its acceptable URL length, appending the %20HTTP/1.1 each time. Chrome and forks seem to ignore it.
the curl output is identical until:
|< HTTP/1.1 301 Moved Permanently
|< Content-length: 0
-|< Location: https://example.com/
+|< Location: https://example.com/ HTTP/1.1
where we can see under 19b6 the HTTP/1.1 sneaks in.
And as we are using the -L follow option in curl we see the incorrect URL being propagated back:
|* Connection #0 to host logs.example.com left intact
-|* Issue another request to this URL: 'https://example.com/'
+|* Issue another request to this URL: 'https://example.com/ HTTP/1.1'
|* Trying 95.216.20.215...
I've tried fiddling with my 301 settings in haproxy.conf to no avail.
environment:
FreeBSD 11.2Rp4 amd64
internet -> haproxy :443 (ipv4 or ipv6) -> h2o 2.3.0b1 backend for serving actual files.
# /usr/local/etc/haproxy/haproxy.conf
... blah
bind ipv4@:443 ssl alpn h2, crt ...
bind ipv6@:443 ssl alpn h2, crt ...
# redirect anything that doesn't match our ACLs or isn't TLS
http-request redirect code 301 location https://example.com%[capture.req.uri] unless www or api or beta
http-request redirect scheme https code 301 if !{ ssl_fc }
I can provide full actual logs / configs off-list if needed, and it's quick to switch in versions of haproxy for validating or get pcap traces. Just bug me on irc dch in #haproxy channel.
haproxy -vvv
HA-Proxy version 1.9-dev6 2018/11/11
*** or HA-Proxy version 1.8.14-52e4d43 2018/09/20
Copyright 2000-2018 Willy Tarreau <***@haproxy.org>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.5
Built with OpenSSL version : OpenSSL 1.0.2o-freebsd 27 Mar 2018
Running on OpenSSL version : OpenSSL 1.0.2o-freebsd 27 Mar 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available multiplexer protocols :
(protocols markes as <default> cannot be specified using 'proto' keyword)
<default> : mode=TCP|HTTP side=FE|BE
h2 : mode=HTTP side=FE
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
patdiff of curl output between http/1.1 and 1.8 vs http/1.1 and 1.9b6:
------ h11_18.log
++++++ h11_19b6.log
@|-39,16 +39,16 ============================================================
|> User-Agent: curl/7.62.0
|> Accept: */*
|>
|{ [5 bytes data]
|< HTTP/1.1 301 Moved Permanently
|< Content-length: 0
!|< Location: https://example.com/ HTTP/1.1
|<
|* Connection #0 to host logs.example.com left intact
-|* Issue another request to this URL: 'https://example.com/'
+|* Issue another request to this URL: 'https://example.com/ HTTP/1.1'
|* Trying 95.216.20.215...
|* TCP_NODELAY set
|* Connected to example.com (95.216.20.215) port 443 (#1)
|* ALPN, offering http/1.1
|* successfully set certificate verify locations:
|* CAfile: /usr/local/share/certs/ca-root-nss.crt
@|-79,36 +79,25 ============================================================
|* start date: Sep 6 19:45:18 2018 GMT
|* expire date: Dec 5 19:45:18 2018 GMT
|* subjectAltName: host "example.com" matched cert's "example.com"
|* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
|* SSL certificate verify ok.
|} [5 bytes data]
!|> GET / HTTP/1.1 HTTP/1.1
|> Host: example.com
|> User-Agent: curl/7.62.0
|> Accept: */*
|>
|{ [5 bytes data]
-|< HTTP/1.1 200 OK
-|< Content-Length: 11297
-|< Server: h2o/2.3.0-beta1
-|< date: Thu, 15 Nov 2018 01:17:58 GMT
-|< content-type: text/html; charset=utf-8
-|< last-modified: Thu, 20 Sep 2018 21:29:23 GMT
-|< etag: "5ba41133-2c21"
-|< vary: accept-encoding
-|< accept-ranges: bytes
-|< content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline' https://cloud.typogra
-|< X-koans: i09
-|< X-sni: example.com
-|< x-powered-by: FreeBSD
-|< x-frame-options: deny
-|< X-XSS-Protection: 1;mode=block
-|< X-Content-Type-Options: nosniff
-|< X-UA-Compatible: IE=Edge
-|< Referrer-Policy: strict-origin-when-cross-origin
-|< Access-Control-Allow-Origin: example.com
-|< Strict-Transport-Security: max-age=15552000
-|< Cache-Control: no-transform
+|* HTTP 1.0, assume close after body
+|< HTTP/1.0 400 Bad request
+|< Cache-Control: no-cache
+|< Connection: close
+|< Content-Type: text/html
|<
-|{ [11297 bytes data]
-|* Connection #1 to host example.com left intact
+|{ [90 bytes data]
+|* TLSv1.2 (IN), TLS alert, close notify (256):
+|{ [2 bytes data]
+|* Closing connection 1
+|} [5 bytes data]
+|* TLSv1.2 (OUT), TLS alert, close notify (256):
+|} [2 bytes data]
thanks
Dave