Discussion:
[ANNOUNCE] haproxy-1.9-dev3
Willy Tarreau
2018-09-29 18:41:03 UTC
Permalink
Subject: [ANNOUNCE] haproxy-1.9-dev3
To: ***@formilux.org

Hi,

Now that Kernel Recipes is over (it was another awesome edition), I'm back
to my haproxy activities. Well, I was pleased to see that my coworkers
reserved me a nice surprise by fixing the pending bugs that were plaguing
dev2. I should go to conferences more often, maybe it's a message from
them to make me understand I'm disturbing them when I'm at the office ;-)

So I thought that it was a good opportunity to issue dev3 now and make it
what dev2 should have been, and forget that miserable one, eventhough I
was told that I'll soon get another batch of patches to merge, but then
we'll simply emit dev4 so there's no need to further delay pending fixes.

HAProxy 1.9-dev3 was released on 2018/09/29. It added 35 new commits
after version 1.9-dev2.

There's nothing fancy here. The connection issues are supposedly addressed
(please expect a bit more in this area soon). The HTTP/1 generic parser is
getting smarter since we're reimplementing the features that were in the
old HTTP code (content-length and transfer-encoding now handled). Lua now
can access stick-tables. I haven't checked precisely how but I saw that
Adis updated the doc so all info should be there.

Ah, a small change is that we now build with -Wextra after having addressed
all warnings reported up to gcc 7.3 and filtered a few useless ones. If you
get some build warnings, please report them along with your gcc version and
your build options. I personally build with -Werror in addition to this one,
and would like to keep this principle to catch certain bugs or new compiler
jokes earlier in the future.

As usual, this is an early development version. It's fine if you want to
test the changes, but avoid putting this into production if it can cost
you your job!

Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Sources : http://www.haproxy.org/download/1.9/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/1.9/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
Adis Nezirovic (1):
MEDIUM: lua: Add stick table support for Lua.

Bertrand Jacquin (1):
DOC: Fix typos in lua documentation

Christopher Faulet (3):
MINOR: h1: Add H1_MF_XFER_LEN flag
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed
BUG/MEDIUM: http: Don't parse chunked body if there is no input data

Dragan Dosen (1):
BUG/MEDIUM: patterns: fix possible double free when reloading a pattern list

Moemen MHEDHBI (1):
DOC: Update configuration doc about the maximum number of stick counters.

Olivier Houchard (4):
BUG/MEDIUM: process_stream: Don't use si_cs_io_cb() in process_stream().
MINOR: h2/stream_interface: Reintroduce te wake() method.
BUG/MEDIUM: h2: Wake the task instead of calling h2_recv()/h2_process().
BUG/MEDIUM: process_stream(): Don't wake the task if no new data was received.

Willy Tarreau (24):
BUG/MINOR: h1: don't consider the status for each header
MINOR: h1: report in the h1m struct if the HTTP version is 1.1 or above
MINOR: h1: parse the Connection header field
MINOR: http: add http_hdr_del() to remove a header from a list
MINOR: h1: add headers to the list after controls, not before
MEDIUM: h1: better handle transfer-encoding vs content-length
MEDIUM: h1: deduplicate the content-length header
CLEANUP/CONTRIB: hpack: remove some h1 build warnings
BUG/MINOR: tools: fix set_net_port() / set_host_port() on IPv4
BUG/MINOR: cli: make sure the "getsock" command is only called on connections
MINOR: stktable: provide an unchecked version of stktable_data_ptr()
MINOR: stream-int: make si_appctx() never fail
BUILD: ssl_sock: remove build warnings on potential null-derefs
BUILD: stats: remove build warnings on potential null-derefs
BUILD: stream: address null-deref build warnings at -Wextra
BUILD: http: address a couple of null-deref warnings at -Wextra
BUILD: log: silent build warnings due to unchecked __objt_{server,applet}
BUILD: dns: fix null-deref build warning at -Wextra
BUILD: checks: silence a null-deref build warning at -Wextra
BUILD: connection: silence a couple of null-deref build warnings at -Wextra
BUILD: backend: fix 3 build warnings related to null-deref at -Wextra
BUILD: sockpair: silence a build warning at -Wextra
BUILD: build with -Wextra and sort out certain warnings
BUG/CRITICAL: hpack: fix improper sign check on the header index value

---
Dmitry Sivachenko
2018-09-29 20:05:19 UTC
Permalink
Post by Willy Tarreau
Ah, a small change is that we now build with -Wextra after having addressed
all warnings reported up to gcc 7.3 and filtered a few useless ones.
Hello,

here are some warnings from clang version 6.0.0:

cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS -DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/cfgparse.o src/cfgparse.c
src/cfgparse.c:5131:34: warning: implicit conversion from 'int' to 'char' changes value from 130 to -126 [-Wconstant-conversion]
curproxy->check_req[5] = 130;
~ ^~~
src/cfgparse.c:5157:33: warning: implicit conversion from 'int' to 'char' changes value from 128 to -128 [-Wconstant-conversion]
curproxy->check_req[5] = 128;
~ ^~~


cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS -DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/stick_table.o src/stick_table.c
src/stick_table.c:2018:14: warning: equality comparison with extraneous parentheses [-Wparentheses-equality]
if ((stkctr == &tmpstkctr))
~~~~~~~^~~~~~~~~~~~~
src/stick_table.c:2018:14: note: remove extraneous parentheses around the comparison to silence this warning
if ((stkctr == &tmpstkctr))
~ ^ ~
src/stick_table.c:2018:14: note: use '=' to turn this equality comparison into an assignment
if ((stkctr == &tmpstkctr))
^~


cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS -DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/mux_h2.o src/mux_h2.c
src/mux_h2.c:3532:195: warning: implicit conversion from enumeration type 'enum h1m_state' to different enumeration type 'enum h1_state' [-Wenum-conversion]
...= %d bytes out (%u in, st=%s, ep=%u, es=%s, h2cws=%d h2sws=%d) data=%u", h2c->st0, h2s->id, size+9, (unsigned int)total, h1_msg_state_str(h1m->state), h1m->err_pos, h1_ms...
~~~~~~~~~~~~~~~~ ~~~~~^~~~~


cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS -DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/peers.o src/peers.c
src/peers.c:253:16: warning: implicit conversion from 'int' to 'char' changes value from 133 to -123 [-Wconstant-conversion]
*msg_type = PEER_MSG_STKT_UPDATE_TIMED;
~ ^~~~~~~~~~~~~~~~~~~~~~~~~~
src/peers.c:255:16: warning: implicit conversion from 'int' to 'char' changes value from 134 to -122 [-Wconstant-conversion]
*msg_type = PEER_MSG_STKT_INCUPDATE_TIMED;
~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/peers.c:259:16: warning: implicit conversion from 'int' to 'char' changes value from 128 to -128 [-Wconstant-conversion]
*msg_type = PEER_MSG_STKT_UPDATE;
~ ^~~~~~~~~~~~~~~~~~~~
src/peers.c:261:16: warning: implicit conversion from 'int' to 'char' changes value from 129 to -127 [-Wconstant-conversion]
*msg_type = PEER_MSG_STKT_INCUPDATE;
~ ^~~~~~~~~~~~~~~~~~~~~~~
src/peers.c:448:11: warning: implicit conversion from 'int' to 'char' changes value from 130 to -126 [-Wconstant-conversion]
msg[1] = PEER_MSG_STKT_DEFINE;
~ ^~~~~~~~~~~~~~~~~~~~
src/peers.c:484:11: warning: implicit conversion from 'int' to 'char' changes value from 132 to -124 [-Wconstant-conversion]
msg[1] = PEER_MSG_STKT_ACK;
~ ^~~~~~~~~~~~~~~~~
Willy Tarreau
2018-09-29 20:35:15 UTC
Permalink
Hi Dmitry,
Post by Dmitry Sivachenko
Post by Willy Tarreau
Ah, a small change is that we now build with -Wextra after having addressed
all warnings reported up to gcc 7.3 and filtered a few useless ones.
Hello,
cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS -DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/cfgparse.o src/cfgparse.c
src/cfgparse.c:5131:34: warning: implicit conversion from 'int' to 'char' changes value from 130 to -126 [-Wconstant-conversion]
curproxy->check_req[5] = 130;
(...)

Oh yes I remember seeing these ones reported once, as well as in peers.c.
Post by Dmitry Sivachenko
src/stick_table.c:2018:14: warning: equality comparison with extraneous parentheses [-Wparentheses-equality]
if ((stkctr == &tmpstkctr))
~~~~~~~^~~~~~~~~~~~~
src/stick_table.c:2018:14: note: remove extraneous parentheses around the comparison to silence this warning
if ((stkctr == &tmpstkctr))
~ ^ ~
src/stick_table.c:2018:14: note: use '=' to turn this equality comparison into an assignment
if ((stkctr == &tmpstkctr))
^~
Interesting ones! I admit that without context around they clearly look
suspicious, though we tend to know this code works. These ones are easy
to fix.

Thank you!
Willy
Willy Tarreau
2018-09-30 09:05:15 UTC
Permalink
Hi Aleks,
Do you have any release date for 1.9, as I plan to launch some new site and
thought to use 1.9 from beginning because it sounds like that 1.9 will be able
to handle h2 with the backend.
It's initially planned for end of October/early November, but I think we'll
stretch the months a little bit. The extremely difficult part is the rework
of the HTTP engine to migrate to the native internal representation which
is needed to transport H2 semantics from end to end. While a huge amount
of work has been done on this, it also uncovered some very old design
heritage that needed to be replaced and that takes time to address, such
as the changes to logging and error snapshots to make them work out of
streams, or the change of connection orientation which we initially expected
to postpone after 1.9 but that we discovered late is mandatory to finish the
work, and the change of the idle connections that's needed to maintain
keep-alive on the backend side.

These changes have a huge impact on the code and the architecture, so as
per the technical vs functional release cycle, I'd really want to have
this in 1.9 so that we have all the basis for much cleaner and calmer
development for 2.0. But I'm sure we will face yet more surprises.

Thus if we see that it's definitely not workable to complete these changes
by ~November, we'll possibly release without them but will put all of them
in a -next branch that we'll merge soon after the release. However if we
manage to have something almost done, I'm willing to push the deadline a
little bit further to let this be finished. Christopher suggested that we
might have a 3rd option which is to have the two implementations side by
side and that we decide by configuration which one to use depending on
the desired features. That's indeed an option (a temporary one) but I
don't like it much due to the risk of increased complexity with bug
reports. That's still definitely something to keep in mind anyway.

I sincerely hope it's the last time we engage in such complex changes in a
single version! I got caught several years ago during the 1.5 development
and this time it's even more complex than what we had to redesign by then!

Hoping this clarifies the situation a bit,
Willy
Tim Düsterhus
2018-10-01 14:44:38 UTC
Permalink
Willy,
Post by Willy Tarreau
Ah, a small change is that we now build with -Wextra after having addressed
all warnings reported up to gcc 7.3 and filtered a few useless ones. If you
get some build warnings, please report them along with your gcc version and
your build options. I personally build with -Werror in addition to this one,
and would like to keep this principle to catch certain bugs or new compiler
jokes earlier in the future.
make -j4 all TARGET=linux2628 USE_SYSTEMD=1 USE_OPENSSL=1 USE_LUA=1
Post by Willy Tarreau
ii gcc 4:8.2.0-1 amd64 GNU C compiler
ii gcc-8 8.2.0-7 amd64 GNU C compiler
ii gcc-8-base:amd64 8.2.0-7 amd64 GCC, the GNU Compiler Collection (base package)
gcc (Debian 8.2.0-7) 8.2.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-
Post by Willy Tarreau
src/ssl_sock.c:7030:32: error: potential null pointer dereference [-Werror=null-dereference]
cipher = SSL_CIPHER_find(conn->xprt_ctx, bin);
~~~~^~~~~~~~~~
src/hlua.c:2365:31: error: potential null pointer dereference [-Werror=null-dereference]
if (appctx->ctx.hlua_cosocket.connected) {
~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
src/hlua.c:2442:15: error: potential null pointer dereference [-Werror=null-dereference]
conn->target = s->target;
~~~~~~~~~~~~~^~~~~~~~~~~
src/hlua.c:2481:2: error: potential null pointer dereference [-Werror=null-dereference]
appctx_wakeup(appctx);
^~~~~~~~~~~~~~~~~~~~~
src/ssl_sock.c:2114:4: error: potential null pointer dereference [-Werror=null-dereference]
s = objt_listener(conn->target)->bind_conf;
~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make: *** [Makefile:929: src/hlua.o] Error 1
make: *** Waiting for unfinished jobs....
cc1: all warnings being treated as errors
make: *** [Makefile:929: src/ssl_sock.o] Error 1
Best regards
Tim Düsterhus
Willy Tarreau
2018-10-15 11:27:13 UTC
Permalink
Hi Tim,
Post by Willy Tarreau
ii gcc 4:8.2.0-1 amd64 GNU C compiler
ii gcc-8 8.2.0-7 amd64 GNU C compiler
ii gcc-8-base:amd64 8.2.0-7 amd64 GCC, the GNU Compiler Collection (base package)
gcc (Debian 8.2.0-7) 8.2.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
(...)

A bit late, but thanks for these warnings, I think all of them are
addressed now.

Willy

Loading...