Oh wow this is really a good time to get the hands dirty as QUIC is a major
design change in HTTP, IMHO.
Some first approaches were already attempted about one year ago already,
to avoid being later in the party. But just like when we had to work on
H2, trying to stuff this into an existing stream-based component is not
easy and we ended up identifying a lot of lower layers that had to be
changed first, so this work was paused and the protocol changed a lot
during that time.
To adopt the matrix from one of the last message, haproxy will be able to do the
conversion in almost every direction?
HTTP/1.x <> HTTP/2
HTTP/2 <> HTTP/3
HTTP/1.x <> HTTP/3
HTTP/3 <> HTTP/3
Yes, that's the idea. In fact it will be slightly different, we're implementing
protocol conversion at the lower layers, to an internal version-agnostic HTTP
representation supposed to accurately represent a message while keeping its
semantics, that we've called HTX. I know that you love ASCII art, so it will
look more or less like this (we'll have to provide a lot of doc but first we
really want to focus on getting the code merged) :
+---------------------------------------+ stream
| all HTTP processing | layer
+---------------------------------------+
^ ^ ^ ^
HTX | HTX | HTX | HTX | normalised
v v v v interface
+------+ +--------+ +--------+ +--------+
|applet| | HTTP/1 | | HTTP/2 | | HTTP/3 | whatever layer (called mux for now
+------+ +--------+ +--------+ +--------+ but may change once we have others,
cache | | | | | could be presentation in OSI)
stats | +----------+ | +------+
Lua svc | | TLS | | | QUIC | transport layer
| +----------+ | +------+
| | | |
+---------------------+ +-----+
| TCP/Unix/socketpair | | UDP | control layer
+---------------------+ +-----+
| |
+----------------------+
| file descriptor | socket layer
+----------------------+
|
+-----------+
| operating |
| system |
+-----------+
It's really over-simplified since several layers above in fact have multiple
upper arrows as they are multiplexed, but it's to explain more or less how
stuff gets stacked. And since there are always transition periods, you have
multiple protocols possible between each layer, otherwise it wouldn't be
fun. You have coded in the very old version where the file descriptors
were directly present in the stream layer. As you can see, over a decade
a lot of new layers have been built between the operating system and the
streams, without even losing features nor compatibility. That's where the
real challenges are.
Is this technically possible as the UDP/QUIC may have not some information that
the TCP/HTTP need?
People tend to confuse QUIC and UDP too much. QUIC more or less uses
UDP as a replacement for IP, which doesn't require privileges, and it
reimplements its own transport stack on top of it, with congestion
control, multiplexing, encryption etc. So it effectively represents a
reliable transport layer for multiple streams based on datagrams below.
UDP is seen as the revolutionary thing when you tell someone that HTTP
will work over UDP but that's absolutely not true, it's almost comparable
to saying that HTTP works over UDP when you're doing it over TLS over
TCP over OpenVPN over UDP.
As I try to imagine the new design in HAProxy will it look like this?
TCP (TCP|UDP) HTTP/1.x,HTTP/2,HTTP/3
\ /
HTTP/TCP/UDP Engine - (TCP|UDP) HTTP/1.x,HTTP/2,HTTP/3
(Here happen some magic)
/ \
UDP (TCP|UDP) HTTP/1.x,HTTP/2,HTTP/3
Really 8-O ?
Precisely not, see above ;-)
What happens with other protocols like plain TCP/UDP or grpc and so on?
Nothing, no particular change, that's the purpose of the stack you see
above. In fact from the info gathered, gRPC will only require that our
H2 mux supports trailers, which were not needed till they unearthed them.
But I want to see end-to-end H2 before working on trailers, as trailers
alone are useless, but H2 end-to-end is useful!
Sorry a lot of questions, but you know I'm curious and try to understand some
parts of the future ;-).
Yes I know, and you're welcome! It's important to discuss design but I
don't want to bore people with it, so it's easier if I have to answer
questions ;-)
Post by Willy TarreauWe still have a lot of work to be done before supporting QUIC but we've
started to have a clear idea how that will fit. The only thing is that
experience taught us the the devil is in the details, and haproxy has
accumulated a lot of details over the years.
Oh yes as always, this devil lay there for years and suddenly he jumps out and
say "nana I'm here" 8-O.
It's even worse. Sometimes we contemplate the thing and start to ask "hey,
doesn't this look like a devil? Maybe we should destroy it right now", and
afterwards we think "sadly, this devil was still slightly useful, now we
have to repair even more than expected". Stay tuned!
Cheers,
Willy