[ANNOUNCE] haproxy-1.7-dev6

[Willy Tarreau]

Hi,

HAProxy 1.7-dev6 was released on 2016/11/09. It added 61 new commits
after version 1.7-dev5.

I must say I'm *really* happy because we managed to merge all the stuff
that was still pending in dev5 and all persons involved managed to
assign some time to get their stuff merged, which I do appreciate given
that we're all pretty busy at this period of the year.

There are still quite some important changes but with limited impact
on existing code since most of these changes were performed in side
areas. The 4 main changes are :

- ability to start a server whose address doesn't resolve, to decide how
it must resolve upon startup, and to let the DNS or the CLI set its IP
address later. This is achieved thanks to the new "init-addr" server
setting. This implies that server address resolution is now delayed to a
later moment in the boot sequence and that it will now be possible to
pre-configure pools of unused servers that can be populated at run time
when needed. This brings extra interesting improvements that we didn't
think about. The first one is that when a config has errors, now you get
all resolution errors at once instead of having to edit the file one
line at a time and to try again. The second one is that it's now trivial
to completely disable server address resolution upon failure, so we
added a new debug option for this (-dr). That's convenient for people
who, like me, often face configs which don't resolve in their
environment and still want to validate the parsing. Please refer to the
doc for this. [work done by Baptiste and me]

- a DNS resolution failure can now finally bring a server down once the
hold time is expired. This has been missing in 1.6 and had for
consequence that traffic could be sent to a wrong server if the address
was reassigned to someone else. This combined with init-addr above will
provide an interesting method to transparently enable/disable servers in
dynamic farms. [work done by Baptiste]

- initial support for OpenSSL 1.1.0 was added. It builds with some warnings
remining that parts of the old API are now deprecated, but it seems to
work. Compatibility with OpenSSL 1.0.1/1.0.2 was maintained and assured
via a compatibility file mapping the new API to the old one. At this
moment, OpenSSL 0.9.8 doesn't build anymore. It doesn't seem terribly
complicated to fix but as usual in this situations it's a painful
process and we preferred to focus on the other pending stuff given that
0.9.8 is not supported anymore. However if someone is willing to address
this, patches are more than welcome! I suggest to add a distinct section
in the openssl-compat file for 0.9.8 as its API differs from 1.0.x. Distro
maintainers might be interested in giving it a try on their next distros.
[work done by Dirkjan Bussink]

- and the new stream processing offload engine (SPOE). Yes, we had to give
it a name. And the protocol is called SPOP. This is what allows haproxy
to offload some of its processing to external processes which can apply
some actions and set variables. There are a few things that really
please me here. The first one obviously is that it was completed in
time. Kudos to Christopher on this one! The next one is that I
personally find the design quite clean and we left some room to improve
the protocol later if needed, and to improve our first implementation of
the protocol without breaking backwards compatibility. The next one is
that the code lies in its own file without affecting the code at all, it
solely relies on the new filters infrastructure, which at the same time
starts to proves its maturity, and this is great. The last one is that
there's quite an extensive doc and even an example of external agent to
be used as a starting point to move your processing outside. Most likely
the first use cases will be to implement various forms of authentication
or content inspection. We're obviously interested in feedback here.
Those not using it don't have to fear any side effect. More info here :

http://www.haproxy.org/download/1.7/doc/SPOE.txt


We also now have a third device detection engine, WURFL, contributed by
Scientiamobile. The code is clean and well isolated so it was not a problem
to merge it this late in the release process. I took this opportunity to
clean up our README by moving the parts specific to DeviceAtlas and
51Degrees to their own file as well because they used to represent 1/3 of
the whole file.

Aside this we fixed the last pending bugs around the systemd wrapper and
the issue I introduced in 1.6 when porting the peers to the new applet
subsystem causing some connections to stay there forever and prevening
old processes from disappearing sometimes upon reload. The drain state
is now properly restored upon reload when using the state-file.

Finally some minor performance improvements were brought to the HTTP parser
for large requests or responses (eg: long URLs, huge cookies). I've observed
up to 10% increase in request rate with 1kB cookies and 100-char URIs.

The goal now really is to test this version and to release it with minimal
changes in 1-2 weeks depending on feedback and bug reports. Yes that's short,
so if you have a few minor pending patches that you'd like to get merged in
1.7, send them NOW. There are still a number of things I'd like to see better
arranged, so cleanups and code moves may still happen, and still are welcome,
but we must not perform other important changes now. Please if you want to
touch anything in dumpstats.c, notify William who is trying to tidy all this
horrible mess by moving all non-stats parts to their relevant files (no code
change, just functions being reshuffled around).

#############################################################################################
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Sources : http://www.haproxy.org/download/1.7/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/1.7/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Happy testing and thanks to participants!
Willy
---
Complete changelog :
- DOC: fix the entry for hash-balance-factor config option
- DOC: Fix typo in description of `-st` parameter in man page
- CLEANUP: cfgparse: Very minor spelling correction
- MINOR: examples: Update haproxy.spec URLs to haproxy.org
- BUG/MEDIUM: peers: on shutdown, wake up the appctx, not the stream
- BUG/MEDIUM: peers: fix use after free in peer_session_create()
- MINOR: peers: make peer_session_forceshutdown() use the appctx and not the stream
- MINOR: peers: remove the pointer to the stream
- BUG/MEDIUM: systemd-wrapper: return correct exit codes
- DOC: stats: provide state details for show servers state
- MEDIUM: tools: make str2ip2() preserve existing ports
- CLEANUP: tools: make ipcpy() preserve the original port
- OPTIM: http: move all http character classs tables into a single one
- OPTIM: http: improve parsing performance of long header lines
- OPTIM: http: improve parsing performance of long URIs
- OPTIM: http: optimize lookup of comma and quote in header values
- BUG/MEDIUM: srv-state: properly restore the DRAIN state
- BUG/MINOR: srv-state: allow to have both CMAINT and FDRAIN flags
- MINOR: server: do not emit warnings/logs/alerts on server state changes at boot
- BUG/MEDIUM: servers: properly propagate the maintenance states during startup
- MEDIUM: wurfl: add Scientiamobile WURFL device detection module
- DOC: move the device detection modules documentation to their own files
- CLEANUP: wurfl: reduce exposure in the rest of the code
- MEDIUM: ssl: Add support for OpenSSL 1.1.0
- MINOR: stream: make option contstats usable again
- MEDIUM: tools: make str2sa_range() return the FQDN even when not resolving
- MINOR: init: move apply_server_state in haproxy.c before MODE_CHECK
- MAJOR: server: postpone address resolution
- MINOR: new srv_admin flag: SRV_ADMF_RMAINT
- MINOR: server: indicate in the logs when RMAINT is cleared
- MINOR: stats: indicate it when a server is down due to resolution
- MINOR: server: make srv_set_admin_state() capable of telling why this happens
- MINOR: dns: implement extra 'hold' timers.
- MAJOR: dns: runtime resolution can change server admin state
- MEDIUM: cli: leave the RMAINT state when setting an IP address on the CLI
- MEDIUM: server: add a new init-addr server line setting
- MEDIUM: server: make use of init-addr
- MINOR: server: implement init-addr none
- MEDIUM: server: make libc resolution failure non-fatal
- MINOR: server: add support for explicit numeric address in init-addr
- DOC: add some documentation for the "init-addr" server keyword
- MINOR: init: add -dr to ignore server address resolution failures
- MEDIUM: server: do not restrict anymore usage of IP address from the state file
- BUG: vars: Fix 'set-var' converter because of a typo
- CLEANUP: remove last references to 'ruleset' section
- MEDIUM: filters: Add attch/detach and stream_set_backend callbacks
- MINOR: filters: Update filters documentation accordingly to recent changes
- MINOR: filters: Call stream_set_backend callbacks before updating backend stats
- MINOR: filters: Remove backend filters attached to a stream only for HTTP streams
- MINOR: flt_trace: Add hexdump option to dump forwarded data
- MINOR: cfgparse: Add functions to backup and restore registered sections
- MINOR: cfgparse: Parse scope lines and save the last one parsed
- REORG: sample: move code to release a sample expression in sample.c
- MINOR: vars: Allow '.' in variable names
- MINOR: vars: Add vars_set_by_name_ifexist function
- MEDIUM: vars: Add a per-process scope for variables
- MINOR: vars: Add 'unset-var' action/converter
- MAJOR: spoe: Add an experimental Stream Processing Offload Engine
- MINOR: spoe: add random ip-reputation service as SPOA example
- MINOR: spoe/checks: Add support for SPOP health checks
- DOC: update ROADMAP file
---

[Willy Tarreau]

Hi Aleks,

Post by Willy Tarreau
http://www.haproxy.org/download/1.7/doc/SPOE.txt

I have read the doc. very interesting.
When I understand this sentence right currently it is only possible to check
some headers right?
###
Actually, for now, the SPOE can offload the processing before "tcp-request
content",
"tcp-response content", "http-request" and "http-response" rules.
###

In theory since you pass sample fetch results as arguments, it should
be possible to pass anything. For example you can already parse the
beginning of a body in http-request if you have enabled the correct
option to wait for the body (http-buffer-request or something like
this). So in theory you could even pass a part of it even right now.

So a header only WAF is now "easily" possible instead of the full stack with
mod_security.
http://blog.haproxy.com/2012/10/12/scalable-waf-protection-with-haproxy-and-apache-with-modsecurity/

In theory yes. And that's one of the goals. My initial intent regarding
this protocol was to be able to delegate some heavy processing outside
of the process, to do it for blocking stuff (eg: ldap libs are always
at list a little bit blocking), as well as for anything requiring
threads.

Then I realized that it would solve other problems. For example we have
3 device detection engines, none of them is ever built in by default
because they have external dependencies, so users who want to use them
have to rebuild haproxy and will not be able to use their distro packages
anymore. Such components could possibly be moved to external agents.

Another point is WAF. People have a love and hate relation with their
WAF, whatever it is. When you deploy your first WAF, you start by loving
it because you see in the logs that it blocks a lot of stuff. Then your
customers complain about breakage and you have to tune it and find a
tradeoff between protection and compatibility. And one day they get
hacked and they declare this WAF useless and you want to change
everything. Having the WAF built into haproxy would mean that users
would have to switch to another LB just to use a different WAF! With
SPOP we can imagine having various WAF implementations in external
processes that users can chose from.

A last motive is stability. At haptech we have implemented support for
loadable modules (and you know how much I don't want to see this in our
version here). Developing these modules require extreme care and lots
of skills regarding haproxy's internals. We currently have a few such
modules, providing nice improvements but whose usage will be debatable
depending on users. Thus supporting modules is interesting because not
everyone is forced to load some code they don't necessarily need or
want, and it saves us from having to maintain patches. However we have
to enforce a very tight check on the internal API to ensure a module is
not loaded on a different version, which means that users have to update
their modules at the same time they update the haproxy executable. But
despite this there's always the risk that a bug in some experimental
code we put there corrupts the whole process and does nasty stuff
(random crashes, corrupted responses, never-ending connections, etc).
With an external process it's much easier for anyone to develop
experimental code without taking any risk for the main process. And if
something crashes, you know on which side it crashes thus you can guess
why. And typically a WAF is not something I would like to see implemented
as a module, I would fear support escalations for crashed processes!

So you see, there are plenty of good reasons for being able to move some
content processing outside of haproxy, and these reasons have driven the
protocol design. The first implementation focuses on having something
usable even if not optimal first (eg: we didn't implement pipelining of
requests yet but given there are connection pools it's not a big deal).

Some attacks are also in the post body, I assume this will come in the
future after some good tests.

Yes that's planned. You should already be able to pass a full buffer of
data using req.body (not tested). This is even why the protocol supports
fragmented payload. It's more complicated to implement though. We could
even imagine doing some compression outside (eg: sdch, snappy, or
whatever). In 1.7, the compression was moved to filters so it's pretty
possible to move it to an external process as well.

We'll be very interested in getting feedback such as "I tried to implement
this and failed". The protocol currently looks nice and evolutive. But I
know by experience that you can plan everything and the first feature
someone requests cannot be fulfilled and will require a protocol update :-)

Post by Willy Tarreau
Finally some minor performance improvements were brought to the HTTP parser
for large requests or responses (eg: long URLs, huge cookies). I've observed
up to 10% increase in request rate with 1kB cookies and 100-char URIs.

For me very impressive, wow respect.

Well for me it's not impressive, these were low hanging fruits that I
identified a long time ago in the parser and which I thought "if I don't
do these micro-optimizations now I'll never do them".

(...)

As I interpret this right the HTTP/2 will be on the roadmap of 1.8 or 2.0?

Yes definitely. But I know too well that a community-driven project cannot
have a roadmap, just hints for contributors. Also I am a bottleneck because
while I review and integrate code I cannot develop. I thought I would have
H2 in 1.6 if you remember :-) So let's say that we'll put a lot of efforts
into getting H2 in 1.8. I also count a lot on SPOP to help move some
contribs outside of the core and to release some of my time :-)

Some of our customers want to use http2_push.
I think this requires that also the HTTP/2 client (Backend) need to be
implemented right?

In theory yes. I don't think we'll have the backend side in 1.8 though
but we'll see. However there is currently a discussion on the HTTP WG
to use 1xx status codes to let an HTTP/1 server pass hints to a client
to retrieve extra objects. These may be usable as well by gateways to
produce H2 push responses to H2 clients. So maybe by then we'd have
support for something like this, I don't know either. Time will tell.

Cheers,
Willy