Discussion:
[ANNOUNCE] haproxy-1.9-dev4
Willy Tarreau
2018-10-21 19:05:24 UTC
Permalink
Hi,

HAProxy 1.9-dev4 was released on 2018/10/21. It added 97 new commits
after version 1.9-dev3.

There's not much user-visible here, it's mostly another merge of some
pending infrastructure changes. The most sensitive changes consist in
the finalization of the connection reorientation from top to bottom,
so that we don't need the conn_want_* tricks from the upper layers nor
the update_poll() calls anymore. Everything is attempted directly and
a subscription to the lower layer is made in case of failure. The perf
is slightly better than with dev3, but more importantly the code becomes
much cleaner and straightforward. An optimization was made in the
scheduler regarding the wait queues, most of which are lockfree now.
Another one concerns the FD lock which is taken less often as well.
All in all the overall multi-thread performance has increased quite
a bit. I measured a gain of 60% over 1.8 using only H2 on 4 threads.

A number of bugs in various areas were addressed (master-worker, rdv
point, h2, streams).

Some preliminary changes to the HTTP/1 engine were made. One of them
concerns the connection header processing which revealed that some
absurd cases were supported in the configuration, and were possibly
working as expected for some users... depending on what they were
expecting. These ones involved "option http-pretend-keepalive" in
frontends and "option http-tunnel" in backends, both of which make
no sense. These ones will now be ignored and will emit a warning if
encountered.

The support for TLS 1.3 ciphersuites was merged. If you play with it,
please report successes or failures, as this was backported to 1.8.
Regarding TLS, certificates can now be generated on the fly on
BoringSSL as well.

Build reports about warnings were all handled, and we tried to address
all of them on gcc 3.4, 4.4, 4.7, 5.3, 6.4, 7.2, 8.1, as well as Clang
3.4, 3.8 and a very recent one that I forgot. The build now looks OK
both on Linux and FreeBSD so that it's possible to add -Werror. By the
way I wanted to add a Makefile option to enable -Werror easily and I
forgot, it will be for later. Again, if you get some warnings, feel
free to pass them on.

Ah, I almost forgot, on systems featuring clock_gettime() with both
CLOCK_THREAD_CPUTIME_ID and CLOCK_MONOTONIC (ie Linux >= 2.6.12 and
maybe other OSes), "show activity" will indicate the number of
milliseconds of CPU that were stolen to each thread by other processes
or threads running on the machine. Normally this indicates improperly
bound threads being parasited by something else. This is expected to
help when some users observe very abnormal performance patterns when
using threads.

Regarding the next steps, we're not that bad in the end, thanks to
the efforts of all participants. From what I've seen hopefully that
next week we'll be able to take a first round of the new native HTTP
representation in experimental state with some limitations. We should
probably have a nice update on the master-worker model, and some
updates on the cache. If we manage to get all this in good shape and
merged soon, I'm willing to push the release date a bit further and
have a longer stabilization phase so that we release 1.9 in better
shape than 1.8, hoping that post-release bugs will have less impact
and will not require as much energy to address.

As usual, this development version is mostly aimed at developers, but
I'm starting to be tempted to deploy it just to test it further, though
I'll possibly do it with dev5 next week. Please don't put it on sensitive
production yet if you want to play with it, as the finalization of the
connection changes have already uncovered some old nasty corner cases,
it could likely trigger a few other ones.

Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Sources : http://www.haproxy.org/download/1.9/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/1.9/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Have fun,
Willy
---
Complete changelog :
Bertrand Jacquin (1):
DOC: Fix a few typos

Christopher Faulet (10):
MINOR: http: Move comment about some HTTP macros in the right header file
MINOR: stats: Add missing include
MINOR: http: Export some functions and do cleanup to prepare HTTP refactoring
MEDIUM: http: Ignore http-pretend-keepalive option on frontend
MEDIUM: http: Ignore http-tunnel option on backend
MINOR: http: Use same flag for httpclose and forceclose options
MINOR: h1: Add EOH marker during headers parsing
MINOR: conn-stream: Add CL_FL_NOT_FIRST flag
MINOR: h1: Change the union h1_sl to use indirect strings to store infos
MINOR: h1: Add the flag H1_MF_NO_PHDR to not add pseudo-headers during parsing

Dirkjan Bussink (4):
MEDIUM: ssl: add support for ciphersuites option for TLSv1.3
CLEANUP: haproxy: Remove unused variable
CLEANUP: h1: Fix debug warnings for h1 headers
CLEANUP: stick-tables: Remove unneeded double (()) around conditional clause

Emeric Brun (2):
BUG/MEDIUM: Cur/CumSslConns counters not threadsafe.
BUG/MEDIUM: mworker: segfault receiving SIGUSR1 followed by SIGTERM.

Emmanuel Hocdet (2):
MINOR: ssl: cleanup old openssl API call
MINOR: ssl: generate-certificates for BoringSSL

Fabrice Fontaine (1):
BUILD: Allow configuration of pcre-config path

Ilya Shipitsin (1):
BUG/MINOR: connection: avoid null pointer dereference in send-proxy-v2

Lukas Tribus (2):
DOC: clarify force-private-cache is an option
DOC: fix reference to map files in MAINTAINERS

Olivier Houchard (19):
BUG/MEDIUM: buffers: Make sure we don't wrap in ci_insert_line2/b_rep_blk.
MINOR: connections: Introduce an unsubscribe method.
MEDIUM: connections: Change struct wait_list to wait_event.
BUG/MEDIUM: h2: Make sure we're not in the send list on flow control.
BUG/MEDIUM: stream: Make sure to unsubscribe before si_release_endpoint.
MINOR: server: Use memcpy() instead of strncpy().
MINOR: build: Disable -Wstringop-overflow.
MINOR: cfgparse: Write 130 as 128 as 0x82 and 0x80.
MINOR: peers: use defines instead of enums to appease clang.
BUG/MEDIUM: pools: Fix the usage of mmap()) with DEBUG_UAF.
BUILD: memory: fix free_list pointer declaration again for atomic CAS
BUG/MEDIUM: h2: Close connection if no stream is left an GOAWAY was sent.
BUG/MEDIUM: connections: Remove subscription if going in idle mode.
BUG/MEDIUM: stream: Make sure polling is right on retry.
MINOR: h2: Make sure to return 1 in h2_recv() when needed.
MEDIUM: connections: Don't directly mess with the polling from the upper layers.
MINOR: streams: Call tasklet_free() after si_release_endpoint().
MINOR: connection: Add a SUB_CALL_UNSUBSCRIBE event.
MINOR: h2: Don't run tasks that are waiting to send if mux in full.

PiBa-NL (1):
REGTEST/MINOR: compatibility: use unix@ instead of abns@ sockets

William Lallemand (1):
BUG/MEDIUM: mworker: don't poll on LI_O_INHERITED listeners

Willy Tarreau (51):
REORG: http: move the code to different files
REORG: http: move HTTP rules parsing to http_rules.c
CLEANUP: http: remove some leftovers from recent cleanups
BUILD: Makefile: add a "make opts" target to simply show the build options
BUILD: Makefile: speed up compiler options detection
BUG/MINOR: backend: check that the mux installed properly
BUG/MEDIUM: h2: check that the connection is still valid at the end of init()
BUG/MEDIUM: h2: make h2_stream_new() return an error on memory allocation failure
MEDIUM: task: perform a single tree lookup per run queue batch
BUG/MINOR: threads: move declaration of capabilities to config.h
OPTIM: tools: optimize my_ffsl() for x86_64
MINOR: log: make sess_log() support sess=NULL
MINOR: chunk: add chunk_cpy() and chunk_cat()
MEDIUM: h2: stop relying on H2_SS_IDLE / H2_SS_CLOSED
CLEANUP: h2: rename h2c_snd_settings() to h2c_send_settings()
MINOR: h2: don't try to send data before preface
MINOR: h2: unify the mux init function
MINOR: h2: retrieve the front proxy from the caller instead of the session
MINOR: h2: split h2c_stream_new() into h2s_new() + h2c_frt_stream_new()
MINOR: h2: add a new flag to quickly distinguish front vs back connection
BUG/MEDIUM: stream: don't crash on out-of-memory
BUILD: compiler: add a new statement "__unreachable()"
BUILD: lua: silence some compiler warnings about potential null derefs
BUILD: ssl: fix null-deref warning in ssl_fc_cipherlist_str sample fetch
BUILD: ssl: fix another null-deref warning in ssl_sock_switchctx_cbk()
BUILD: stick-table: make sure not to fail on task_new() during initialization
BUILD: peers: check allocation error during peers_init_sync()
MINOR: tools: add a new function atleast2() to test masks for more than 1 bit
MINOR: config: use atleast2() instead of my_popcountl() where relevant
MEDIUM: fd/threads: only grab the fd's lock if the FD has more than one thread
MAJOR: tasks: create per-thread wait queues
OPTIM: tasks: group all tree roots per cache line
MINOR: pools: allocate most memory pools from an array
MINOR: pools: split pool_free() in the lockfree variant
MEDIUM: pools: implement a thread-local cache for pool entries
BUG/MEDIUM: threads: fix thread_release() at the end of the rendez-vous point
Revert "BUILD: lua: silence some compiler warnings about potential null derefs"
BUILD: lua: silence some compiler warnings about potential null derefs (#2)
MINOR: lua: all functions calling lua_yieldk() may return
BUILD: lua: silence some compiler warnings after WILL_LJMP
BUILD: Makefile: silence an option conflict warning with clang
CLEANUP: state-file: make the path concatenation code a bit more consistent
MINOR: fd: centralize poll timeout computation in compute_poll_timeout()
MINOR: poller: move time and date computation out of the pollers
BUILD: memory: fix pointer declaration for atomic CAS
BUILD: Makefile: add USE_RT to pass -lrt for clock_gettime() and friends
MINOR: time: add now_mono_time() and now_cpu_time()
MEDIUM: time: measure the time stolen by other threads
BUILD: memory: fix free_list pointer declaration again for atomic CAS
BUILD: compiler: rename __unreachable() to my_unreachable()
MINOR: ebtree: save 8 bytes in struct eb32sc_node

mildis (2):
BUG/MINOR: h2: null-deref
BUG/MINOR: checks: queues null-deref

---
Pavlos Parissis
2018-10-22 09:01:37 UTC
Permalink
Post by Willy Tarreau
Hi,
HAProxy 1.9-dev4 was released on 2018/10/21. It added 97 new commits
after version 1.9-dev3.
There's not much user-visible here, it's mostly another merge of some
pending infrastructure changes. The most sensitive changes consist in
the finalization of the connection reorientation from top to bottom,
so that we don't need the conn_want_* tricks from the upper layers nor
the update_poll() calls anymore. Everything is attempted directly and
a subscription to the lower layer is made in case of failure. The perf
is slightly better than with dev3, but more importantly the code becomes
much cleaner and straightforward. An optimization was made in the
scheduler regarding the wait queues, most of which are lockfree now.
Another one concerns the FD lock which is taken less often as well.
All in all the overall multi-thread performance has increased quite
a bit. I measured a gain of 60% over 1.8 using only H2 on 4 threads.
Nice, very nice. HAProxy version 1.9 would be very exciting release.

Any ideas if we get see gRPC support on both ends(client/server side) in 1.9 version ?

Cheers,
Pavlos
Willy Tarreau
2018-10-22 09:15:55 UTC
Permalink
Hi Pavlos!
Post by Pavlos Parissis
Post by Willy Tarreau
Hi,
HAProxy 1.9-dev4 was released on 2018/10/21. It added 97 new commits
after version 1.9-dev3.
There's not much user-visible here, it's mostly another merge of some
pending infrastructure changes. The most sensitive changes consist in
the finalization of the connection reorientation from top to bottom,
so that we don't need the conn_want_* tricks from the upper layers nor
the update_poll() calls anymore. Everything is attempted directly and
a subscription to the lower layer is made in case of failure. The perf
is slightly better than with dev3, but more importantly the code becomes
much cleaner and straightforward. An optimization was made in the
scheduler regarding the wait queues, most of which are lockfree now.
Another one concerns the FD lock which is taken less often as well.
All in all the overall multi-thread performance has increased quite
a bit. I measured a gain of 60% over 1.8 using only H2 on 4 threads.
Nice, very nice. HAProxy version 1.9 would be very exciting release.
Yes, we all hope so, but we also expect that it may still have rough edges,
being a technical version.
Post by Pavlos Parissis
Any ideas if we get see gRPC support on both ends(client/server side) in 1.9 version ?
While we won't be doing anything specific for gRPC, my previous reading of
the spec taught me that if we simply support H2 end to end it *should* work.
And at the moment, even if I don't want to promise then have to retract, it
looks like we should be able to have at least experimental support for
end-to-end H2 via the new internal HTTP representation. The amount of work
to convert all HTTP-aware functions to the new representation is so huge
that it's almost impossible to do everything in this first version, so if
all goes well, we'll have support for the basic HTTP stuff and probably not
all rulesets, filters, compression, Lua nor anything fancy at the release
date, hence the "experimental" status. But putting that aside, for most
setups it should still be quite sufficient, making it worth being released
anyway.

Cheers,
Willy
Pavlos Parissis
2018-10-23 07:42:26 UTC
Permalink
Post by Willy Tarreau
Hi Pavlos!
Post by Pavlos Parissis
Post by Willy Tarreau
Hi,
HAProxy 1.9-dev4 was released on 2018/10/21. It added 97 new commits
after version 1.9-dev3.
There's not much user-visible here, it's mostly another merge of some
pending infrastructure changes. The most sensitive changes consist in
the finalization of the connection reorientation from top to bottom,
so that we don't need the conn_want_* tricks from the upper layers nor
the update_poll() calls anymore. Everything is attempted directly and
a subscription to the lower layer is made in case of failure. The perf
is slightly better than with dev3, but more importantly the code becomes
much cleaner and straightforward. An optimization was made in the
scheduler regarding the wait queues, most of which are lockfree now.
Another one concerns the FD lock which is taken less often as well.
All in all the overall multi-thread performance has increased quite
a bit. I measured a gain of 60% over 1.8 using only H2 on 4 threads.
Nice, very nice. HAProxy version 1.9 would be very exciting release.
Yes, we all hope so, but we also expect that it may still have rough edges,
being a technical version.
Post by Pavlos Parissis
Any ideas if we get see gRPC support on both ends(client/server side) in 1.9 version ?
While we won't be doing anything specific for gRPC, my previous reading of
the spec taught me that if we simply support H2 end to end it *should* work.
So, haproxy wont be able to understand services and calls of those services and take
load balancing decisions in the same way we do with HTTP1 where we can take decisions
base on verbs(HTTP Methods) and nouns(URL, cookie and etc).

I have to mention that I know very little about gRPC/brotobuf and I have looked what
other products offer when they claim that they support gRPC and I didn't find any
other software loadbalancer that is able to route traffic based on the verbs/nouns.
It could be that my expectation is unrealistic and lb software doesn't need to
take any decisions based on those verbs/nouns.

Cheers,
Pavlos
Willy Tarreau
2018-10-23 07:49:11 UTC
Permalink
Hi Pavlos,
Post by Pavlos Parissis
Post by Willy Tarreau
While we won't be doing anything specific for gRPC, my previous reading of
the spec taught me that if we simply support H2 end to end it *should* work.
So, haproxy wont be able to understand services and calls of those services and take
load balancing decisions in the same way we do with HTTP1 where we can take decisions
base on verbs(HTTP Methods) and nouns(URL, cookie and etc).
I'm not sure what you mean because the basic stuff (http actions, fetching
samples, ACLs etc) will be OK. What will definitely not be ready will be to
have a TCP frontend connecting to an HTTP backend, "tcp-request content"
rules processing HTTP elements, Lua, etc.
Post by Pavlos Parissis
I have to mention that I know very little about gRPC/brotobuf and I have looked what
other products offer when they claim that they support gRPC and I didn't find any
other software loadbalancer that is able to route traffic based on the verbs/nouns.
If this is the only thing they need, that sounds fairly basic. What we
will not have will be support for PUSH but this is not required for gRPC
(I initially thought it was because some people speak about bidirectionnal
HTTP/2 when they want to mean in fact HTTP/2 on both sides). That's why
I'm really saying that it's possible that in the end you'll have everything
you need.
Post by Pavlos Parissis
It could be that my expectation is unrealistic and lb software doesn't need to
take any decisions based on those verbs/nouns.
I don't know what is needed, but I definitely expect that at least this
should work :-)

Cheers,
Willy
Willy Tarreau
2018-10-23 08:55:14 UTC
Permalink
Hi Aleks,
Post by Willy Tarreau
I'm not sure what you mean because the basic stuff (http actions, fetching
samples, ACLs etc) will be OK. What will definitely not be ready will be to
have a TCP frontend connecting to an HTTP backend, "tcp-request content"
rules processing HTTP elements, Lua, etc.
Will be h2c possible, as some gRPC server starts with HTTP not HTTPS?
We don't have the H1-to-H2 upgrade at the moment, but 1.9 will support H2
without TLS (just specify "proto h2" on the "bind" or "server" lines).
Post by Willy Tarreau
Post by Pavlos Parissis
It could be that my expectation is unrealistic and lb software doesn't need to
take any decisions based on those verbs/nouns.
I don't know what is needed, but I definitely expect that at least this
should work :-)
I think also but what I have seen out there is hat 'Te: trailers' could be a
challenge, as this blog entry describes.
https://nghttp2.org/blog/2015/03/24/proxying-grpc-with-nghttpx/
I'm not sure if haproxy can handle this.
No we don't have trailers in H2 for now. However the good news is that
it's not because it's that difficult to implement, it's only because
nothing at all ever used them for 2 decades... until gRPC apparently
decided that they deserved more love. So I'm quite confident that we
could have trailers support not too long after 1.9. Thanks for pointing
this, this definitely helps! By the way, Tatsuhiro mentions H1 to H2
proxying in the article above, that's definitely something that should
work as well with haproxy with the new native internal representation.

Cheers,
Willy
Dirkjan Bussink
2018-10-23 12:22:08 UTC
Permalink
Hi all,
Post by Willy Tarreau
The support for TLS 1.3 ciphersuites was merged. If you play with it,
please report successes or failures, as this was backported to 1.8.
Regarding TLS, certificates can now be generated on the fly on
BoringSSL as well.
Is there any guess on a timeline for the next 1.8 release? Would love to play with the new TLS 1.3 options there then and test those out in our staging environments.

Cheers,

Dirkjan
Willy Tarreau
2018-10-23 12:48:45 UTC
Permalink
Hi Dirkjan,
Post by Dirkjan Bussink
Hi all,
Post by Willy Tarreau
The support for TLS 1.3 ciphersuites was merged. If you play with it,
please report successes or failures, as this was backported to 1.8.
Regarding TLS, certificates can now be generated on the fly on
BoringSSL as well.
Is there any guess on a timeline for the next 1.8 release? Would love to play
with the new TLS 1.3 options there then and test those out in our staging
environments.
You're right. I started backporting fixes for it last week. I think it
would make sense to make one "soon" (maybe next week-end along dev5).
In the mean time you can pick the latest maintenance snapshot if you
want, it already contains your work.

Cheers,
Willy
Dirkjan Bussink
2018-11-20 15:13:47 UTC
Permalink
Hi Willy,
Post by Willy Tarreau
You're right. I started backporting fixes for it last week. I think it
would make sense to make one "soon" (maybe next week-end along dev5).
In the mean time you can pick the latest maintenance snapshot if you
want, it already contains your work.
Could I ask again for a release of 1.8 as well? 1.9 has seen a few more dev releases already and wondering if 1.8 can get one again?

Cheers,

Dirkjan
Willy Tarreau
2018-11-20 15:19:48 UTC
Permalink
Hi Dirkjan,
Post by Dirkjan Bussink
Hi Willy,
Post by Willy Tarreau
You're right. I started backporting fixes for it last week. I think it
would make sense to make one "soon" (maybe next week-end along dev5).
In the mean time you can pick the latest maintenance snapshot if you
want, it already contains your work.
Could I ask again for a release of 1.8 as well? 1.9 has seen a few more dev
releases already and wondering if 1.8 can get one again?
Indeed it's already been two months, it would be the right time to emit
a new one. But at the moment all the people able to work on this are
fully loaded finishing their respective parts for 1.9 (or fixing it).
Are you missing a specific fix at the moment ? I'm asking because the
1.8 queue doesn't look huge and most of the recently fixed bugs in 1.9
do not affect 1.8.

Regards,
Willy
Dirkjan Bussink
2018-11-22 10:41:11 UTC
Permalink
Hi Willy,
Post by Willy Tarreau
Indeed it's already been two months, it would be the right time to emit
a new one. But at the moment all the people able to work on this are
fully loaded finishing their respective parts for 1.9 (or fixing it).
Are you missing a specific fix at the moment ? I'm asking because the
1.8 queue doesn't look huge and most of the recently fixed bugs in 1.9
do not affect 1.8.
Mostly looking for having my TLS 1.3 work in an official released version.

Cheers,

Dirkjan
Willy Tarreau
2018-11-22 10:55:42 UTC
Permalink
Hi Dirkjan,
Post by Dirkjan Bussink
Hi Willy,
Post by Willy Tarreau
Indeed it's already been two months, it would be the right time to emit
a new one. But at the moment all the people able to work on this are
fully loaded finishing their respective parts for 1.9 (or fixing it).
Are you missing a specific fix at the moment ? I'm asking because the
1.8 queue doesn't look huge and most of the recently fixed bugs in 1.9
do not affect 1.8.
Mostly looking for having my TLS 1.3 work in an official released version.
Ah yes of course, I forgot about this one as I was focusing on bugs only!
This totally makes sense indeed! I'll do my best to make a pause away from
1.9 but it's not easy at the moment.

Thanks!
Willy

Loading...