Willy Tarreau
2018-11-11 10:24:55 UTC
Hi,
HAProxy 1.9-dev6 was released on 2018/11/11. It added 42 new commits
after version 1.9-dev5.
This version comes with a small frustration, as I really wanted to
include Christopher's HTX work and Olivier's changes on the idle
connection management, but I faced a few issues and difficulties
merging them together so I'm keeping this stuff out for now and will
have to use their help next week to sort this out. However I'm still
releasing dev6 because it contains numerous fixes for issues plaguing
the previous versions.
The positive point of this is that when we're dealing with merging
issues, it means that we've got way past the point of agreeing with
the design :-)
Since dev3 or so, we've faced some (expected) erratic behaviours on the
connection and applets sides, as a result of the very tricky changes in
the connection management. One of these issues directly involved a bad
practice we've been having for years around the use of stream-interface
flag SI_FL_WAIT_ROOM which used to convey two signals at once. After the
changes, we could only have either the applets or the connections work
fine but not both :-/ We managed to finally address this using much
more effort, audit and rework than initially imagined, due to deep roots
hanging below this flag. The current code is still far from being clean,
but it fixes the 100% CPU we've been seeing on pending outgoing
connections after the rework. We'll be able to proceed with more cleanups
once we're done with the idle connection changes I guess.
In parallel I noticed that we've broken TCP splicing. Not once, not
twice... At least 5 times since 1.9-dev3! I'm saying "at least" because I
could bisect 4 times after reverting each faulty patch and gave up when
seeing the 5th. I'm not too much worried because the benefit of all these
very difficult changes is that the code becomes more logical and that we
can figure a bit easier what to do at certain places. It's just that we
still have a lot of antique code having fun with our nerves. So please
do not use "option splice-*" for now, it will lead to random connection
freezes.
I can't say that there really are any new features in this version. Most
of the patches address connection issues, recent master-worker issues,
and a few new info on the CLI (show proc and a few fields in show info).
So overall it's mostly a stability fix on top of -dev5 for those willing
to play with it.
If we can manage to sort out the few issues met with idle connection
changes and rebase the htx code on top of that, I'm fine with emitting
a new version early this week. Such changes will provide us with server-
side keep-alive when H2 is used on the front, and with the revamped HTTP
stack that we'll need to provide end-to-end H2.
I also have to review and merge Joseph's cleanups for typos in the doc
(it was too late for this one, being busy dealing with connection issues).
I'm seeing that we're almost done with merging the pending features, and
that's reassuring, as we'll probably have a long testing phase coming.
This week will be the one determining if we can still expect to release
1.9 by the end of the month or not. We've noticed that a lot of cleanup
is still needed (mostly rename certain types, functions or fields that
became completely misleading over time), but that can be done during the
testing phase since it's not supposed to incur any functionality change.
Bug reports are welcome, and as usual, don't put this on sensitive prod
yet!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Sources : http://www.haproxy.org/download/1.9/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/1.9/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
---
Complete changelog :
Christopher Faulet (2):
MEDIUM: stream-int: Rely only on SI_FL_WAIT_ROOM to stop data receipt
MEDIUM: stream-int: Try to read data even if channel's buffer seems to be full
Cyril Bonté (2):
DOC: split the http-request actions in their own section
DOC: split the http-response actions in their own section
Lukas Tribus (1):
DOC: restore note about "independant" typo
William Lallemand (10):
BUG/MINOR: cli: forward the whole command on master CLI
MINOR: cli: show the number of reload in 'show proc'
MINOR: cli: can't connect to the target CLI
MEDIUM: mworker: does not create the CLI proxy when no listener
MINOR: mworker: displays more information when leaving
MEDIUM: mworker: exit with the incriminated exit code
MINOR: mworker: displays a message when a worker is forked
MEDIUM: mworker: leave when the master die
BUG/MEDIUM: mworker: does not abort() in mworker_pipe_register()
BUG/MEDIUM: cli: crash when trying to access a worker
Willy Tarreau (27):
BUG/MEDIUM: tools: fix direction of my_ffsl()
BUG/MEDIUM: auth/threads: use of crypt() is not thread-safe
MINOR: compat: automatically detect support for crypt_r()
MEDIUM: auth/threads: make use of crypt_r() on systems supporting it
BUG/MAJOR: stream-int: don't call si_cs_recv() in stream_int_chk_rcv_conn()
BUG/MINOR: tasks: make sure wakeup events are properly reported to subscribers
MINOR: stats: report the number of active jobs and listeners in "show info"
MINOR: stats: report the number of active peers in "show info"
MINOR: stats: report the number of currently connected peers
CLEANUP: stream-int: retro-document si_cs_io_cb()
BUG/MEDIUM: stream-int: don't wake up for nothing during SI_ST_CON
MEDIUM: stream: implement stream_buf_available()
MEDIUM: appctx: check for allocation attempts in buffer allocation callbacks
MINOR: stream-int: rename si_applet_{want|stop|cant}_{get|put}
MINOR: stream-int: add si_done_{get,put} to indicate that we won't do it anymore
MINOR: stream-int: use si_cant_put() instead of setting SI_FL_WAIT_ROOM
MINOR: stream-int: make use of si_done_{get,put}() in shut{w,r}
MINOR: stream-int: make it clear that si_ops cannot be null
MEDIUM: stream-int: temporarily make si_chk_rcv() take care of SI_FL_WAIT_ROOM
MINOR: stream-int: factor the SI_ST_EST state test into si_chk_rcv()
MEDIUM: stream-int: make SI_FL_WANT_PUT reflect CF_DONT_READ
MEDIUM: stream-int: always call si_chk_rcv() when we make room in the buffer
MEDIUM: stream-int: make si_chk_rcv() check that SI_FL_WAIT_ROOM is cleared
MINOR: stream-int: replace si_update() with si_update_both()
MEDIUM: stream-int: make stream_int_update() aware of the lower layers
CLEANUP: stream-int: remove the now unused si->update() function
BUG/MINOR: config: better detect the presence of the h2 pattern in npn/alpn
---
HAProxy 1.9-dev6 was released on 2018/11/11. It added 42 new commits
after version 1.9-dev5.
This version comes with a small frustration, as I really wanted to
include Christopher's HTX work and Olivier's changes on the idle
connection management, but I faced a few issues and difficulties
merging them together so I'm keeping this stuff out for now and will
have to use their help next week to sort this out. However I'm still
releasing dev6 because it contains numerous fixes for issues plaguing
the previous versions.
The positive point of this is that when we're dealing with merging
issues, it means that we've got way past the point of agreeing with
the design :-)
Since dev3 or so, we've faced some (expected) erratic behaviours on the
connection and applets sides, as a result of the very tricky changes in
the connection management. One of these issues directly involved a bad
practice we've been having for years around the use of stream-interface
flag SI_FL_WAIT_ROOM which used to convey two signals at once. After the
changes, we could only have either the applets or the connections work
fine but not both :-/ We managed to finally address this using much
more effort, audit and rework than initially imagined, due to deep roots
hanging below this flag. The current code is still far from being clean,
but it fixes the 100% CPU we've been seeing on pending outgoing
connections after the rework. We'll be able to proceed with more cleanups
once we're done with the idle connection changes I guess.
In parallel I noticed that we've broken TCP splicing. Not once, not
twice... At least 5 times since 1.9-dev3! I'm saying "at least" because I
could bisect 4 times after reverting each faulty patch and gave up when
seeing the 5th. I'm not too much worried because the benefit of all these
very difficult changes is that the code becomes more logical and that we
can figure a bit easier what to do at certain places. It's just that we
still have a lot of antique code having fun with our nerves. So please
do not use "option splice-*" for now, it will lead to random connection
freezes.
I can't say that there really are any new features in this version. Most
of the patches address connection issues, recent master-worker issues,
and a few new info on the CLI (show proc and a few fields in show info).
So overall it's mostly a stability fix on top of -dev5 for those willing
to play with it.
If we can manage to sort out the few issues met with idle connection
changes and rebase the htx code on top of that, I'm fine with emitting
a new version early this week. Such changes will provide us with server-
side keep-alive when H2 is used on the front, and with the revamped HTTP
stack that we'll need to provide end-to-end H2.
I also have to review and merge Joseph's cleanups for typos in the doc
(it was too late for this one, being busy dealing with connection issues).
I'm seeing that we're almost done with merging the pending features, and
that's reassuring, as we'll probably have a long testing phase coming.
This week will be the one determining if we can still expect to release
1.9 by the end of the month or not. We've noticed that a lot of cleanup
is still needed (mostly rename certain types, functions or fields that
became completely misleading over time), but that can be done during the
testing phase since it's not supposed to incur any functionality change.
Bug reports are welcome, and as usual, don't put this on sensitive prod
yet!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Sources : http://www.haproxy.org/download/1.9/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/1.9/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
---
Complete changelog :
Christopher Faulet (2):
MEDIUM: stream-int: Rely only on SI_FL_WAIT_ROOM to stop data receipt
MEDIUM: stream-int: Try to read data even if channel's buffer seems to be full
Cyril Bonté (2):
DOC: split the http-request actions in their own section
DOC: split the http-response actions in their own section
Lukas Tribus (1):
DOC: restore note about "independant" typo
William Lallemand (10):
BUG/MINOR: cli: forward the whole command on master CLI
MINOR: cli: show the number of reload in 'show proc'
MINOR: cli: can't connect to the target CLI
MEDIUM: mworker: does not create the CLI proxy when no listener
MINOR: mworker: displays more information when leaving
MEDIUM: mworker: exit with the incriminated exit code
MINOR: mworker: displays a message when a worker is forked
MEDIUM: mworker: leave when the master die
BUG/MEDIUM: mworker: does not abort() in mworker_pipe_register()
BUG/MEDIUM: cli: crash when trying to access a worker
Willy Tarreau (27):
BUG/MEDIUM: tools: fix direction of my_ffsl()
BUG/MEDIUM: auth/threads: use of crypt() is not thread-safe
MINOR: compat: automatically detect support for crypt_r()
MEDIUM: auth/threads: make use of crypt_r() on systems supporting it
BUG/MAJOR: stream-int: don't call si_cs_recv() in stream_int_chk_rcv_conn()
BUG/MINOR: tasks: make sure wakeup events are properly reported to subscribers
MINOR: stats: report the number of active jobs and listeners in "show info"
MINOR: stats: report the number of active peers in "show info"
MINOR: stats: report the number of currently connected peers
CLEANUP: stream-int: retro-document si_cs_io_cb()
BUG/MEDIUM: stream-int: don't wake up for nothing during SI_ST_CON
MEDIUM: stream: implement stream_buf_available()
MEDIUM: appctx: check for allocation attempts in buffer allocation callbacks
MINOR: stream-int: rename si_applet_{want|stop|cant}_{get|put}
MINOR: stream-int: add si_done_{get,put} to indicate that we won't do it anymore
MINOR: stream-int: use si_cant_put() instead of setting SI_FL_WAIT_ROOM
MINOR: stream-int: make use of si_done_{get,put}() in shut{w,r}
MINOR: stream-int: make it clear that si_ops cannot be null
MEDIUM: stream-int: temporarily make si_chk_rcv() take care of SI_FL_WAIT_ROOM
MINOR: stream-int: factor the SI_ST_EST state test into si_chk_rcv()
MEDIUM: stream-int: make SI_FL_WANT_PUT reflect CF_DONT_READ
MEDIUM: stream-int: always call si_chk_rcv() when we make room in the buffer
MEDIUM: stream-int: make si_chk_rcv() check that SI_FL_WAIT_ROOM is cleared
MINOR: stream-int: replace si_update() with si_update_both()
MEDIUM: stream-int: make stream_int_update() aware of the lower layers
CLEANUP: stream-int: remove the now unused si->update() function
BUG/MINOR: config: better detect the presence of the h2 pattern in npn/alpn
---