varnishtest with H2>HTX>H1(keep-alive)

[PiBa-NL]

Hi List,

I'm trying (and failing?) to write a H2>HTX>H1(keepalive) test.

Using haproxy 1.9-dev6-05b9b64.

Test vtc attached, i added the 'option http-use-htx' to the fe4
frontend/backend.
Is there anything else that should be changed?
Or is my way of making the H2 request incorrect? Though the 3 tests
before it 'seem' to work alright.

Below the output i get, with a unexpected '500' status, and with a IC--
on the logline... It also seems it never contacted the s4 server.
Without the htx option it does make 1 request to the s4, and the second
expected request tries to make a second connection. (the 'old' way..)

Thanks in advance.

Regards
PiBa-NL (Pieter)

**   c4    0.2 === txreq -req GET -url /3
***  c4    0.2 tx: stream: 1, type: HEADERS (1), flags: 0x05, size: 37
**   c4    0.2 === rxresp
***  h1    0.2 debug|00000007:fe4.accept(000e)=0011 from [::1:25432]
ALPN=<none>
***  h1    0.2 debug|00000007:fe4.clireq[0011:ffffffff]: GET /3 HTTP/1.1
***  h1    0.2 debug|00000007:b4.clicls[0011:adfd]
***  h1    0.2 debug|00000007:b4.closed[0011:adfd]
***  h1    0.2 debug|::1:25432 [19/Nov/2018:01:03:11.550] fe4 b4/srv4
0/0/-1/-1/0 500 203 - - IC-- 2/1/0/0/0 0/0 "GET /3 HTTP/1.1"
***  c4    0.2 rx: stream: 1, type: HEADERS (1), flags: 0x04, size: 26
***  c4    0.2 flag: END_TYPE_HEADERS
**** c4    0.2 header[ 0]: :status : 500
**** c4    0.2 header[ 1]: cache-control : no-cache
**** c4    0.2 header[ 2]: content-type : text/html
***  c4    0.2 rx: stream: 1, type: DATA (0), flags: 0x00, size: 96
***  c4    0.2 rx: stream: 1, type: DATA (0), flags: 0x01, size: 0
***  c4    0.2 flag: END_STREAM
**** c4    0.2 s1 - no data
**   c4    0.2 === expect resp.status == 200
---- c4    0.2 EXPECT resp.status (500) == "200" failed

[Willy Tarreau]

Hi Pieter,

Post by PiBa-NL
Hi List,
I'm trying (and failing?) to write a H2>HTX>H1(keepalive) test.
Using haproxy 1.9-dev6-05b9b64.
Test vtc attached, i added the 'option http-use-htx' to the fe4
frontend/backend.
Is there anything else that should be changed?

For HTX, you need dev7.

Post by PiBa-NL
Or is my way of making the H2 request incorrect? Though the 3 tests before
it 'seem' to work alright.

I've never tested varnishtest on h2 yet, I don't know precisely how
it's supposed to be configured, nor what minimum version is required
for this. From an haproxy perspective, your config looks OK.

By the way, in my opinion you should remove "option http-keep-alive"
since it's the default, and I think the config would be more readable
by replacing the "frontend"+"backend" couples with a single "listen"
for each.

Post by PiBa-NL
Below the output i get, with a unexpected '500' status, and with a IC-- on
the logline... It also seems it never contacted the s4 server.

Indeed, it faced an internal while trying to make the connection.
What I think happened is that H2 decoded the request (and logged it),
but the upper layer stream failed to do anything from it since it's
configured with HTX and HTX is not supported in your version. Please
note that even with dev7 you don't have H2 over HTX yet. You
definitely need to update to dev7 to eliminate a number of candidate
bugs.

Post by PiBa-NL
Without the htx option it does make 1 request to the s4, and the second
expected request tries to make a second connection. (the 'old' way..)

Without the latest changes from dev7 it's expecetd since by default,
server-side keep-alive is lost when H2 is used on the frontend (the
connection used to be tied to the stream, so since you have a distinct
stream for each request, you used to lose the connection). In dev7 the
server-side idle connection moved to the session which is attached to
the front connection so the keep-alive will still work.

Hoping this helps,
Willy

[PiBa-NL]

Hi Willy,

Post by Willy Tarreau
Hi Pieter,

Post by PiBa-NL
Hi List,
I'm trying (and failing?) to write a H2>HTX>H1(keepalive) test.
Using haproxy 1.9-dev6-05b9b64.
Test vtc attached, i added the 'option http-use-htx' to the fe4
frontend/backend.
Is there anything else that should be changed?

For HTX, you need dev7.

Ah crap, i 'thought' i took the latest commit from the online branch.
And stopped looking properly.  I must have been a few minutes to soon or
something, after i read the dev7 mail.. (and i should have checked i did
actually compile the expected version..) (which i totally didn't do in
the excitement about htx and the late time..)

Post by Willy Tarreau

Post by PiBa-NL
Or is my way of making the H2 request incorrect? Though the 3 tests before
it 'seem' to work alright.

I've never tested varnishtest on h2 yet, I don't know precisely how
it's supposed to be configured, nor what minimum version is required
for this. From an haproxy perspective, your config looks OK.
By the way, in my opinion you should remove "option http-keep-alive"
since it's the default

That was a remnant of a previous try to get keep-alive with h2 which
then wasn't supposed to work with dev5 yet.

Post by Willy Tarreau
, and I think the config would be more readable
by replacing the "frontend"+"backend" couples with a single "listen"
for each.

Okay true, listen sections would make the config more readable :) . I'm
just used to make a frontend+backend for almost 'everything'.. (Usually
i have multiple backends behind one frontend anyhow..)  And also i
'think' it shows more clearly in the logging output if it did or didn't
get passed from frontend to the backend, but maybe thats just my
imagination.

Post by Willy Tarreau

Post by PiBa-NL
Below the output i get, with a unexpected '500' status, and with a IC-- on
the logline... It also seems it never contacted the s4 server.

Indeed, it faced an internal while trying to make the connection.
What I think happened is that H2 decoded the request (and logged it),
but the upper layer stream failed to do anything from it since it's
configured with HTX and HTX is not supported in your version. Please
note that even with dev7 you don't have H2 over HTX yet. You
definitely need to update to dev7 to eliminate a number of candidate
bugs.

Post by PiBa-NL
Without the htx option it does make 1 request to the s4, and the second
expected request tries to make a second connection. (the 'old' way..)

Without the latest changes from dev7 it's expecetd since by default,
server-side keep-alive is lost when H2 is used on the frontend (the
connection used to be tied to the stream, so since you have a distinct
stream for each request, you used to lose the connection). In dev7 the
server-side idle connection moved to the session which is attached to
the front connection so the keep-alive will still work.

Okay new try with "1.9-dev7-1611965".

Post by Willy Tarreau
Hoping this helps,
Willy

However that still doesn't work yet (as also already seen by Frederic):

**   c4    0.2 === txreq -req GET -url /3
***  c4    0.2 tx: stream: 1, type: HEADERS (1), flags: 0x05, size: 37
**   c4    0.2 === rxresp
***  h1    0.2 debug|00000007:fe4.accept(000e)=0010 from [::1:13402]
ALPN=<none>
**** h1    0.2 STDOUT poll 0x11
***  c4    0.2 HTTP2 rx EOF (fd:6 read: No error: 0)
---- c4    0.2 could not get frame header
**   c4    0.2 Ending stream 1
***  c4    0.2 closing fd 6
**   c4    0.2 Ending
*    top   0.2 RESETTING after ./PB-TEST/h2-keepalive-backend.vtc
**   h1    0.2 Reset and free h1 haproxy 31909
**   h1    0.2 Wait
**   h1    0.2 Stop HAproxy pid=31909
**   h1    0.2 WAIT4 pid=31909 status=0x008b (user 0.013928 sys 0.000000)
*    h1    0.2 Expected exit: 0x0 signal: 0 core: 0
---- h1    0.2 Bad exit status: 0x008b exit 0x0 signal 11 core 128
*    top   0.2 failure during reset
#    top  TEST ./PB-TEST/h2-keepalive-backend.vtc FAILED (0.169) exit=2
***@freebsd11:/usr/ports/net/haproxy-devel # haproxy -v
HA-Proxy version 1.9-dev7-1611965 2018/11/19
Copyright 2000-2018 Willy Tarreau <***@haproxy.org>

So i guess the question remains, is the test configured wrongly, or is
some other improvement still needed?
(I guess improvement in this case surely is needed as crashing is never
the right way, even if the input might be 'wrong'.)

Regards,
PiBa-NL (Pieter)

[Willy Tarreau]

Ah crap, i 'thought' i took the latest commit from the online branch. And
stopped looking properly.  I must have been a few minutes to soon or
something, after i read the dev7 mail.. (and i should have checked i did
actually compile the expected version..) (which i totally didn't do in the
excitement about htx and the late time..)

Hehe :-)

Post by Willy Tarreau
, and I think the config would be more readable
by replacing the "frontend"+"backend" couples with a single "listen"
for each.

Okay true, listen sections would make the config more readable :) . I'm just
used to make a frontend+backend for almost 'everything'.. (Usually i have
multiple backends behind one frontend anyhow..)  And also i 'think' it shows
more clearly in the logging output if it did or didn't get passed from
frontend to the backend, but maybe thats just my imagination.

No it's not your imagination, that's a good point indeed. The config is
a bit less readable for tests, but I agree that logs are more precise.

---- h1    0.2 Bad exit status: 0x008b exit 0x0 signal 11 core 128

Wow, segv, it seems we still have some issues left to be addressed,
thank you :-)

So i guess the question remains, is the test configured wrongly, or is some
other improvement still needed?
(I guess improvement in this case surely is needed as crashing is never the
right way, even if the input might be 'wrong'.)

We definitely need to understand and fix the crash first before figuring
how to improve the test, if needed at all.

Thanks!
Willy

[PiBa-NL]

Hi Christopher, Willy,

Hi,
The H2 is not yet compatible with the HTX for now. So you should never
use both in same time. However, this configuration error should be
detected during the configuration parsing, to avoid runtime errors.
Here is a patch to do so. I'll merge it.
Thanks
--
Christopher Faulet

Thanks the 'old' config which tried to combine H2 and HTX is now
rejected. (as expected)

I guess i misinterpreted the 'need' for HTX for the H2 conversion and
features which i thought would include the new keep-alive this version
brings, but i guess those are separate things. Keepalive for a H1
backend coming from a H2 frontend works fine without using that option.

New testcase attached, one that actually works! , regarding H2 > H1 with
keepalive. (without HTX option though..) It shows as 'passed' when run.

I did notice there is one line regarding the 'double logging' I have got
configured though which I'm not sure is supposed to happen, its seems to
be because i'm having both stdout and :514 logging should that not be
possible?:
***  h1    0.0 debug|[ALERT] 323/233813 (57777) : sendmsg()/writev()
failed in logger #2: Socket operation on non-socket (errno=38)

Partial config:
  global
    log stdout format raw daemon
    log :1514 local0

I'm using "HA-Proxy version 1.9-dev7-7ff4f14 2018/11/20" this time.

Or is it (again) something i'm configuring wrongly ? ;) .

Regards,
PiBa-NL (Pieter)

[Willy Tarreau]

Hi Pieter,

Post by PiBa-NL
I did notice there is one line regarding the 'double logging' I have got
configured though which I'm not sure is supposed to happen, its seems to be
because i'm having both stdout and :514 logging should that not be
***  h1    0.0 debug|[ALERT] 323/233813 (57777) : sendmsg()/writev() failed
in logger #2: Socket operation on non-socket (errno=38)
  global
    log stdout format raw daemon
    log :1514 local0

I'm not sure what you mean with "double logging". Above you're asking to
send the log both to stdout and to local UDP port 1514. The sendmsg/writev
error very likely is caused by the UDP one if nothing is listening locally.
But I'm a bit surprised by the error since it's supposed to be a socket.
I'll try to reproduce in case I messed up with the FDs when implementing
this.

Thanks!
Willy