Discussion:
Client timeout expiring while waiting for server response
Francois Rajotte
2016-03-04 17:21:59 UTC
Permalink
Hi,

I'm having an issue with the client timeout expiring unexpectedly.

I am using haproxy v1.5.14 with the http-keep-alive option.

In my use case, I have a client maintaining a long-lived connection through
haproxy to a backend server. The backend server sends back a response 30
seconds after receiving a request, so I set the server timeout at 60
seconds to be safe. I'm keeping the client timeout at 5 seconds.

What I'm observing is the following:
The client establishes a connection and sends a request.
30 seconds later, the server responds.
Right after receiving the response, the client sends a new request through
the same connection.
5 seconds later, the client timeout expires. In the logs, I see cD-- as the
error code.

When I increase the client timeout to above 30 seconds, then the client
timeout no longer expires.

It was my belief that once the client sends a request, haproxy should no
longer use the client timeout, since as the documentation mentions:

The inactivity timeout applies when the client is expected to acknowledge or
send data.

Since the client has sent its request, it is not expected to send data
and since the server has not sent back anything, the client isn't
expected to acknowledge any data either.

It seems to me that the behavior is correct for the first request on
the connection, but incorrect for the following requests on the same
connection.

For the record, the requests I'm using are not using the "Connection:
keep-alive" headers and the responses are not using the "Connection:
close" header either.

I understand that the documentation specifically mentions that both
the client and server timeouts should be kept equal, but I still don't
understand why I'm seeing client timeouts while waiting for the server
response.

Could someone enlighten me about the inner workings of the client timeout?

Thanks,

François
--
This email is intended solely for the person or entity to which it is
addressed and may contain confidential and/or privileged information. If
you are not the intended recipient and have received this email in error,
please notify BroadSoft, Inc. immediately by replying to this message, and
destroy all copies of this message, along with any attachment, prior to
reading, distributing or copying it.
Lukas Tribus
2016-03-04 17:37:13 UTC
Permalink
Hi Francois,
Post by Francois Rajotte
Hi,
I'm having an issue with the client timeout expiring unexpectedly.
I am using haproxy v1.5.14 with the http-keep-alive option.
In my use case, I have a client maintaining a long-lived connection
through haproxy to a backend server. The backend server sends back a
response 30 seconds after receiving a request, so I set the server
timeout at 60 seconds to be safe. I'm keeping the client timeout at 5
seconds.
The client establishes a connection and sends a request.
30 seconds later, the server responds.
Right after receiving the response, the client sends a new request
through the same connection.
5 seconds later, the client timeout expires. In the logs, I see cD-- as
the error code.
When I increase the client timeout to above 30 seconds, then the client
timeout no longer expires.
Did you configure "timeout http-keep-alive"? What if you set that
to 30 seconds and leave the client timeout at 5 seconds?

Do you have a tcpdump capture of the frontend traffic?


Regards,

Lukas
Francois Rajotte
2016-03-04 17:55:46 UTC
Permalink
Hi Lukas,

I did try increasing "timeout http-keep-alive", without any effect.

I'm attaching a tcpdump from the client side. For this capture I had the
client timeout at 10 seconds rather than 5, but the effect is the same.
You can also see that, even though the client timeout triggered, the server
was allowed to send a final response before the connection closed.

For your information, 10.9.35.100 is the client, 10.9.35.32 is haproxy and
10.9.35.34 is the backend.

I'm also attaching a tcpdump of both frontend and backend traffic, this
time with a client timeout of 5 seconds.

Regards,
François
Post by Lukas Tribus
Hi Francois,
Post by Francois Rajotte
Hi,
I'm having an issue with the client timeout expiring unexpectedly.
I am using haproxy v1.5.14 with the http-keep-alive option.
In my use case, I have a client maintaining a long-lived connection
through haproxy to a backend server. The backend server sends back a
response 30 seconds after receiving a request, so I set the server
timeout at 60 seconds to be safe. I'm keeping the client timeout at 5
seconds.
The client establishes a connection and sends a request.
30 seconds later, the server responds.
Right after receiving the response, the client sends a new request
through the same connection.
5 seconds later, the client timeout expires. In the logs, I see cD-- as
the error code.
When I increase the client timeout to above 30 seconds, then the client
timeout no longer expires.
Did you configure "timeout http-keep-alive"? What if you set that
to 30 seconds and leave the client timeout at 5 seconds?
Do you have a tcpdump capture of the frontend traffic?
Regards,
Lukas
--
This email is intended solely for the person or entity to which it is
addressed and may contain confidential and/or privileged information. If
you are not the intended recipient and have received this email in error,
please notify BroadSoft, Inc. immediately by replying to this message, and
destroy all copies of this message, along with any attachment, prior to
reading, distributing or copying it.
Cyril Bonté
2016-03-06 21:49:29 UTC
Permalink
Hi all,

I didn't have much time to look at this issue, but I made a quick test
this evening.
Post by Francois Rajotte
Hi Lukas,
I did try increasing "timeout http-keep-alive", without any effect.
I'm attaching a tcpdump from the client side. For this capture I had the
client timeout at 10 seconds rather than 5, but the effect is the same.
You can also see that, even though the client timeout triggered, the
server was allowed to send a final response before the connection closed.
For your information, 10.9.35.100 is the client, 10.9.35.32 is haproxy
and 10.9.35.34 is the backend.
I'm also attaching a tcpdump of both frontend and backend traffic, this
time with a client timeout of 5 seconds.
I'm not sure to reproduce exactly the same bug with the same
configuration, but I think this has been fixed with commit 3de8e7a
"BUG/MEDIUM: http: don't enable auto-close on the response side" :

http://www.haproxy.org/git?p=haproxy-1.5.git;a=commit;h=3de8e7ab8d9125402cc1a8fb48ee475ee21d7d4c

François, are you able to make a test with the latest 1.5 snapshot ?
--
Cyril Bonté
Francois Rajotte
2016-03-07 14:09:46 UTC
Permalink
Hi Cyril,

I confirm the problem is fixed with the latest 1.5 snapshot. Do you have an
idea when I can expect the next 1.5 minor release to be available?

For your information, here is the build I used:

HA-Proxy version 1.5.15-ce0162e 2016/02/25
Copyright 2000-2015 Willy Tarreau <***@haproxy.org>

Build options :
TARGET = linux26
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
USE_STATIC_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1q 3 Dec 2015
Running on OpenSSL version : OpenSSL 1.0.1h 5 Jun 2014 (VERSIONS DIFFER!)
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.37 2015-04-28
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.


Thanks again for your help!
François
--
This email is intended solely for the person or entity to which it is
addressed and may contain confidential and/or privileged information. If
you are not the intended recipient and have received this email in error,
please notify BroadSoft, Inc. immediately by replying to this message, and
destroy all copies of this message, along with any attachment, prior to
reading, distributing or copying it.
Lukas Tribus
2016-03-07 16:08:24 UTC
Permalink
Hi,
Post by Francois Rajotte
Hi Cyril,
I confirm the problem is fixed with the latest 1.5 snapshot. Do you
have an idea when I can expect the next 1.5 minor release to be
available?
You could upgrade to 1.6.3 which is stable and contains this fix, but
I do agree that we could drop a 1.5.16 (CC'ing Willy):

~/haproxy-1.5$ git log --oneline v1.5.15.. | grep BUG | wc -l
22



Lukas

Loading...