1.7.11 with gzip compression serves incomplete files

[Veiko Kukk]

Hi!

There is not much to add, just that it's been broken before in 1.7.9 and
is again broken in 1.7.11. Works with 1.7.10.
When applying patch provided here
https://www.mail-archive.com/***@formilux.org/msg27155.html 1.7.11
also works.
Testing is really simple, just configure haproxy gzip compression and
download with curl --compression or with web browser. Sample .js file I
have downloaded has real size 42202 bytes but when downloading with gzip
compression, it's size is 37648 bytes - part of the end is missing.
Very similar issue is discussed here too
https://discourse.haproxy.org/t/1-7-11-compression-issue-parsing-errors-on-response/2542

Best regards,
Veiko

[Willy Tarreau]

Hi Veiko,

Hi!
There is not much to add, just that it's been broken before in 1.7.9 and is
again broken in 1.7.11. Works with 1.7.10.
When applying patch provided here
works.
Testing is really simple, just configure haproxy gzip compression and
download with curl --compression or with web browser. Sample .js file I have
downloaded has real size 42202 bytes but when downloading with gzip
compression, it's size is 37648 bytes - part of the end is missing.
Very similar issue is discussed here too https://discourse.haproxy.org/t/1-7-11-compression-issue-parsing-errors-on-response/2542

Interesting, this patch was indeed never merged but a slightly different
one with the exact same date and a slightly different description was
merged instead. Thus I'm CCing Christopher to get his opinion on this.
We likely have to merge it all the way back to 1.7, but I want to be
extra careful.

Thank you very much for the report and for testing the patch!

Willy

[Veiko Kukk]

Now, I'm still puzzled with this issue. Because I can't reproduce it
for now. And it is even more strange because when the compression is
enabled and used on a response, it cannot be switched in TUNNEL mode.
So I don't really understand how the patch you mentioned could fix a
compression bug, or the commit 8066ccd39 (as stated on discourse)
could be the origin of the bug.

I've found that 'option http-server-close' in frontend is causing this.
Commenting it out and 1.7.11 is working fine with gzip compression.

I'm not gathering more debug data at the moment, maybe this already
helps to reproduce the issue. I will provide more if necessary.

Finally, I have a last question. You said the result is truncated. It
means the response is truncated because of a close and not all chunks
are received ? Or the response is correct from the HTTP point of view,
but the file is truncated once uncompressed ?

Uncompressed file is truncated, part of it at the end is missing.
Not sure what do you mean by correctness from HTTP point of view, but
the headers look fine.

GET /assets/js/piwik.js?v=1534 HTTP/1.1
Host: foobar.tld
User-Agent: curl/7.61.1
Accept: */*
Accept-Encoding: deflate, gzip, br

{ [5 bytes data]
< HTTP/1.1 200 OK
< Date: Wed, 05 Dec 2018 15:45:26 GMT
< Content-Encoding: gzip
< Content-Language: en
< Content-Location: http://foobar.tld/assets/js/piwik.js?v=1534
< Content-Type: application/x-javascript; charset=UTF-8
< Expires: Sat, 02 Dec 2028 15:45:26 GMT
< Last-Modified: Fri, 23 Nov 2018 15:51:00 GMT
< Cache-Control: max-age=315360000
< Date: Wed, 05 Dec 2018 15:45:26 GMT
< Accept-Ranges: bytes
< Server: Restlet-Framework/2.3.9
< Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
< Connection: close

65236 Dec 5 16:29 piwik.js (without option http-server-close)
41440 Dec 5 17:45 piwik.js_ (with option http-server-close)

Regards,
Veiko

[Willy Tarreau]

Hi Veiko,

Post by Veiko Kukk

Now, I'm still puzzled with this issue. Because I can't reproduce it
for now. And it is even more strange because when the compression is
enabled and used on a response, it cannot be switched in TUNNEL mode.
So I don't really understand how the patch you mentioned could fix a
compression bug, or the commit 8066ccd39 (as stated on discourse)
could be the origin of the bug.

I've found that 'option http-server-close' in frontend is causing this.
Commenting it out and 1.7.11 is working fine with gzip compression.
I'm not gathering more debug data at the moment, maybe this already helps to
reproduce the issue. I will provide more if necessary.

This is already quite helpful, thank you for these details.

Post by Veiko Kukk

Finally, I have a last question. You said the result is truncated. It
means the response is truncated because of a close and not all chunks
are received ? Or the response is correct from the HTTP point of view,
but the file is truncated once uncompressed ?

Uncompressed file is truncated, part of it at the end is missing.
Not sure what do you mean by correctness from HTTP point of view, but the
headers look fine.

GET /assets/js/piwik.js?v=1534 HTTP/1.1
Host: foobar.tld
User-Agent: curl/7.61.1
Accept: */*
Accept-Encoding: deflate, gzip, br

{ [5 bytes data]
< HTTP/1.1 200 OK
< Date: Wed, 05 Dec 2018 15:45:26 GMT
< Content-Encoding: gzip
< Content-Language: en
< Content-Location: http://foobar.tld/assets/js/piwik.js?v=1534
< Content-Type: application/x-javascript; charset=UTF-8
< Expires: Sat, 02 Dec 2028 15:45:26 GMT
< Last-Modified: Fri, 23 Nov 2018 15:51:00 GMT
< Cache-Control: max-age=315360000
< Date: Wed, 05 Dec 2018 15:45:26 GMT
< Accept-Ranges: bytes
< Server: Restlet-Framework/2.3.9
< Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
< Connection: close
65236 Dec 5 16:29 piwik.js (without option http-server-close)
41440 Dec 5 17:45 piwik.js_ (with option http-server-close)

OK so it looks like the last block is missing. Maybe we lose these
contents when closing the server's connection. Also it's useful to
see that it happens on small files, I expected that it would only
fail on very large ones when the output buffer is full, and it's
not the case here.

I must confess that most of the people able to look closer to
this are currently busy trying to finalize 1.9 for an imminent
release, so we'll have to postpone this debugging a little bit.
In the mean time it would be useful to see if adding
"option http-pretend-keepalive" helps. This way we'll know if
it's the server closing first or haproxy closing first which
triggers this. And if it turns out that it fixes the issue for
you, it could be a good temporary workaround.

Cheers,
Willy

[Veiko Kukk]

Hi, Willy

Post by Willy Tarreau
In the mean time it would be useful to see if adding
"option http-pretend-keepalive" helps. This way we'll know if
it's the server closing first or haproxy closing first which
triggers this. And if it turns out that it fixes the issue for
you, it could be a good temporary workaround.

Indeed, adding "option http-pretend-keepalive" helps.

Veiko

[Willy Tarreau]

Post by Veiko Kukk
Hi, Willy

Post by Willy Tarreau
In the mean time it would be useful to see if adding
"option http-pretend-keepalive" helps. This way we'll know if
it's the server closing first or haproxy closing first which
triggers this. And if it turns out that it fixes the issue for
you, it could be a good temporary workaround.

Indeed, adding "option http-pretend-keepalive" helps.

Great, thanks! So this means that the issue is with the close received
form the server and not with the close sent to the server. This helps
narrow the issue down. If you could take a network capture between
haproxy and the server on a connection exhibiting the problem, it would
be awesome. I'm saying this because I feel like there could be one
behaviour explaining this : if we tell your server that we'd like it
to close at the end of the transfer, and that for whatever reason it
closes in a dirty manner (for example before reading the whole request
in case of a POST), then it could cause it to send an RST and destroy
the last few segments in flight that have not yet been received. This
case is unusual so it's unlikely to be this but it could be an
explanation and could justify why Christopher couldn't reproduce it
last time.

Thanks,
Willy