reg-test failures on FreeBSD, how to best adapt/skip some tests?

[PiBa-NL]

Hi List,

I was wondering how to best run the reg-tests that are 'valid' for FreeBSD.

There are a 2 tests that use abns@ sockets, which seem not available on
FreeBSD.
Also 1 test is failing for a reason i'm not totally sure if its totally
expected to or not..

- /connection/b00000.vtc
probably does not 'really' need abns@ sockets, so changing to unix@
would make it testable on more platforms?

- /log/b00000.vtc
Not exactly sure why this fails/why it was supposed to work.
It either produces a timeout, or the s1 server fails to read the request
which the tcp-healthcheck does not send..

    ***  s1    0.0 accepted fd 5 127.0.0.1 23986
    **   s1    0.0 === rxreq
    ---- s1    0.0 HTTP rx failed (fd:5 read: Connection reset by peer)

- /seamless-reload/b00000.vtc
This one specifically mentions testing a abns@ socket functionality. so
changing it to a unix@ socket likely changes the test in such a way its
no longer testing what it was meant for..
What would be the best way to skip this test on FreeBSD?

With a few small changes (attached) i can run all tests like this, get
the following result:

varnishtest -l ./work/haproxy-ss-20180901/reg-tests/*/*.vtc
#    top  TEST
./work/haproxy-ss-20180901/reg-tests/connection/b00000.vtc passed (0.142)
#    top  TEST ./work/haproxy-ss-20180901/reg-tests/log/b00000.vtc
passed (0.136)
#    top  TEST ./work/haproxy-ss-20180901/reg-tests/lua/b00000.vtc
passed (0.121)
#    top  TEST ./work/haproxy-ss-20180901/reg-tests/lua/b00001.vtc
passed (0.133)
#    top  TEST ./work/haproxy-ss-20180901/reg-tests/lua/b00002.vtc
passed (0.186)
#    top  TEST ./work/haproxy-ss-20180901/reg-tests/lua/b00003.vtc
passed (0.133)
#    top  TEST ./work/haproxy-ss-20180901/reg-tests/lua/h00001.vtc
passed (0.120)
#    top  TEST
./work/haproxy-ss-20180901/reg-tests/seamless-reload/b00000.vtc passed
(0.143)
#    top  TEST ./work/haproxy-ss-20180901/reg-tests/spoe/b00000.vtc
passed (0.011)
#    top  TEST ./work/haproxy-ss-20180901/reg-tests/ssl/b00000.vtc
passed (0.146)
#    top  TEST
./work/haproxy-ss-20180901/reg-tests/stick-table/b00000.vtc passed (0.120)
#    top  TEST
./work/haproxy-ss-20180901/reg-tests/stick-table/b00001.vtc passed (0.122)

Above would be good :).. But well.. needs changing or skipping some tests..

I would like to know next time that 'all' testable reg-tests are working
properly when making a new build to take into my production system. If
some tests fail by design (on this platform), it takes more
administration to figure out if that was okay or not.

Please advice :)

Regards,

PiBa-NL (Pieter)

[PiBa-NL]

Hi Frederic,

Post by PiBa-NL
Hi List,
I was wondering how to best run the reg-tests that are 'valid' for FreeBSD.
on FreeBSD.
Also 1 test is failing for a reason i'm not totally sure if its
totally expected to or not..
- /connection/b00000.vtc
would make it testable on more platforms?
- /log/b00000.vtc
Not exactly sure why this fails/why it was supposed to work.
It either produces a timeout, or the s1 server fails to read the
request which the tcp-healthcheck does not send..
     ***  s1    0.0 accepted fd 5 127.0.0.1 23986
     **   s1    0.0 === rxreq
     ---- s1    0.0 HTTP rx failed (fd:5 read: Connection reset by peer)

This test relies on timeout values to produce the correct syslog output.
Relying on timing values for reg testing is perhaps a bad idea as the
results are not deterministic.
Indeed here we want the 5ms timeout client to expire before it reads
the server response after having waited for 20ms.
I propose to remove this test.

Regarding /log/b00000.vtc  i think the test itself is 'mostly' okay.

If we keep the check i do think its 'check' on the server line should be
changed by one of the following options:
- removing 'check'
- check to different port to a s2 server that does not use txresp
- add 'option httpchk' and repeat s1 twice..
Any of the changes above would 'fix' the biggest issue with the test as
far as i see..

But indeed having short timeouts could cause on occasion cause random
failures if for example multiple tests are run simultaneously..

The test for a decent log line with a 'client timout' flag is probably
something that should be tested, perhaps the c1 delay could be set to
something longer to 'enforce' the 'timeout client 5' would always hit
first even when stuff starts competing for (cpu) resources.?

Regards,
PiBa-NL (Pieter)

[PiBa-NL]

Hi Frederic,

Post by PiBa-NL
- /connection/b00000.vtc
would make it testable on more platforms?

Correct. I agree I did not think to replace this part specific to Linux.

Should i send a patch changing that? Or can you fix it easily ;)

Post by PiBa-NL
- /log/b00000.vtc
Not exactly sure why this fails/why it was supposed to work.
It either produces a timeout, or the s1 server fails to read the
request which the tcp-healthcheck does not send..
     ***  s1    0.0 accepted fd 5 127.0.0.1 23986
     **   s1    0.0 === rxreq
     ---- s1    0.0 HTTP rx failed (fd:5 read: Connection reset by peer)

Perhaps the syslog traces could give us more information about what is
happening here.

Afaik the 'check' on the server line is failing to send a GET /
request.. My other mail has a bit more ideas about that..

Post by PiBa-NL
- /seamless-reload/b00000.vtc
way its no longer testing what it was meant for..
What would be the best way to skip this test on FreeBSD?

Perhaps we should use the TARGET value to select the VTC files
directories which should be selected for the OSes.
By default for linux all VTC files in reg-tests directory should be
run (found with find command without -L option, so that not to follow
the symbolic link).
For instance for freebsd OS we would create reg-tests/freebsd directory
with symbolic links to the linux reg-tests subdirectories it supports.

I think creating a list of tests that could be run on FreeBSD will take
a lot of maintenance and i assume 'most' tests will actually be runnable
on most OS's. So having short list with exceptions is probably easier to
maintain. In my opinion every test should be run on every os unless
there is some good reason not to run it (abns/splicing/stuff..). And if
possible the reason to exclude a specific test should be described (a
single line of text could be enough).
The even bigger issue that Willy raised is with compilation options not
including lua/ssl/gzip/threads/ stuff, and having lots of tests fail on
those, while they should get skipped if they rely on a feature not
compiled. I discussed this also a bit with Willy in the other mail
thread ( https://www.mail-archive.com/***@formilux.org/msg31345.html
), but i don't think we have defined the perfect way to do it yet..

So that part of the question still stands ;) .. Whats the best way to
skip tests that are not applicable?

Regards,
PiBa-NL (Pieter)

[PiBa-NL]

Hi Frederic,

At this time I do not know how to skip the tests which are not supported
due to disable features at compilation time without running a
"haproxy -vv" command before launching the VTC files.

Made a little script (attached)..

It would allow for the .vtc files to contain what 'requirements' they
have, by including lines like these:

#EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd
#REQUIRE_OPTION=OPENSSL, this test needs ssl compiled in.

Which could then generate output like listed below. It searches for all
testcases, validates their requirements, and then puts them in a list.
After which it calls varnishtest with quiet and simultaneous job
parameters to allow for a short turnaround time..

What do you think ? Could this work for current 'requirements'? Or is
something different preferred? I though about running each test directly
from the script, and while that allows per test OK/Fail response logging
for and a nicer overall summery, it takes more time to run. So i went
for automatically creating a list of tests and letting varnishtest
itself handle the running of multiple tests at once.

Thoughts appreciated :).

Regards,

PiBa-NL (Pieter)

***@freebsd11:/usr/ports/net/haproxy-devel # ./run-regtests.sh
########################## Preparing to run tests ##########################
HA-Proxy version 1.8.14-52e4d43 2018/09/20
Copyright 2000-2018 Willy Tarreau <***@haproxy.org>

########################## Gathering tests to run ##########################
Skip ./haproxy_test_OK_20180831/lua/b00002.vtc because option OPENSSL
not found
  REASON:  this test needs ssl compiled in.
Add test: ./haproxy_test_OK_20180831/lua/h00001.vtc
Add test: ./haproxy_test_OK_20180831/lua/b00000.vtc
Add test: ./haproxy_test_OK_20180831/lua/b00001.vtc
Add test: ./haproxy_test_OK_20180831/loadtest/b00000-loadtest.vtc
Add test: ./haproxy_test_OK_20180831/log/b00000.vtc
Skip ./haproxy_test_OK_20180831/reload/b00001-server-state-file.vtc
because: TARGET = freebsd
  REASON:  abns sockets are not available on freebsd
Skip ./haproxy_test_OK_20180831/ssl/b00000.vtc because option OPENSSL
not found
  REASON:  this test needs ssl compiled in.
Add test: ./haproxy_test_OK_20180831/stick-table/b00001.vtc
Add test: ./haproxy_test_OK_20180831/stick-table/b00000.vtc
Add test: ./test/b00000-loadtest.vtc
Skip ./test/b00002.vtc because option OPENSSL not found
  REASON:  this test needs ssl compiled in.
Add test: ./test/b00003-cpu.vtc
Add test: ./work/haproxy-1.8-52e4d43/reg-tests/server/b00000.vtc
Add test: ./work/haproxy-1.8-52e4d43/reg-tests/stick-table/h00000.vtc
Skip ./work/haproxy-1.8-52e4d43/reg-tests/lua/b00002.vtc because option
OPENSSL not found
  REASON:  this test needs ssl compiled in.
Add test: ./work/haproxy-1.8-52e4d43/reg-tests/lua/h00001.vtc
########################## Starting varnishtest ##########################
#    top  TEST ./test/b00003-cpu.vtc FAILED (5.281) exit=2
1 tests failed, 0 tests skipped, 11 tests passed
***@freebsd11:/usr/ports/net/haproxy-devel #

[PiBa-NL]

Hi Frederic,

Do you have a little time to take a look at (the idea of) the script?
Thanks in advance :).

Post by PiBa-NL
Hi Frederic,
Made a little script (attached)..
What do you think ? Could this work for current 'requirements'?

Regards,

PiBa-NL (Pieter)

[PiBa-NL]

Hi Frederic,

The idea of the script sounds good to me. About the script itself it
is a nice work which could be a good start.

Thanks.

Just a few details below.
Note that "cat $file | grep something" may be shortened by "grep
something $file". It would also be interesting to avoid creating
temporary files as most as possible (at least testflist.lst is not
necessary I think).
TARGET=$(haproxy -vv | grep "TARGET  = " | sed 's/.*= //')
OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //')
    { read TARGET; read OPTIONS; } << EOF
    $(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //')
    EOF

Thanks, I've changed this.

which is portable, or something similar.
    sed 's/.*=//'| sed 's/,.*//'
may be shortened by
    sed -e 's/.*=//' -e 's/,.*//'

Thanks, I've changed this as well.

Also note, there are some cases where options are enabled without
appearing in OPTIONS variable value.
   $ make TARGET=linux2628
the support for the thread is enabled without appearing in OPTIONS
variable value. I am not sure this is an issue at this time.

That could become an issue. but should be easy to solve adding a 'fake'
option perhaps to check against.. Or adding a separate check perhaps I'm
not sure yet.

Perhaps we could could only one line for the required options and
#EXCLUDED_TARGETS=freebsd,dos,windows     ;)
#REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB

Added this option, but then i like the option of excluding single
targets and having some comment behind it explaining the reason.. But i
guess if one would want to know some comments above the setting could
also 'explain' why that target is currently not 'valid' to run the test
on. Should i remove the settings for the 'single' option/target.?

New 'version' of the script attached.
It now supports a set of parameters to modify its behavior a little. And
also checking for a 'version requirement'. So a H2 test doesn't have to
fail on 1.7.
Should i 'ask' to delete old test result.? Or would there be a other
better way to keep previous results separated from the current run?
If you could give it another look i would appreciate that. Are there any
things that need to be added/changed about it before considering to add
it to haproxy sources branch?

Regards,

PiBa-NL (Pieter)

[Willy Tarreau]

Hi guys,

Post by PiBa-NL
New 'version' of the script attached.
It now supports a set of parameters to modify its behavior a little. And
also checking for a 'version requirement'. So a H2 test doesn't have to
fail on 1.7.

Very good work.

I missed this message last week (I'm not often on the list these days
I confess), but this tool is awesome!

Post by PiBa-NL
Should i 'ask' to delete old test result.? Or would there be a other
better way to keep previous results separated from the current run?

No.
bname=$(basename $i)
mv /tmp/$bname /tmp/old.$bname
with $i in $(find /tmp/ -type d -name "vtc.*")
or something like that.

It would be nice to support TMPDIR so that we can force it in a script
instead of /tmp. What I *think* could be done by default would be to
delete the old.vtc.* files there before running the tests so that we
always keep only the last results by default. I found hundreds of vtc
directories in /tmp on my laptop and that can become problematic, even
worse on shared development machines.

please *add -type d* to the find command to list the temporary varnishtest
directories.

Well, then add -maxdepth 1 or 2 because you certainly don't want to run
find into my /tmp which contains complete distros to chroot into!

And we should only keep temporaries directory of tests which have failed
with -l varnishtest option currently enabled option. They are all the
/tmp/vtc.* directories found after having run the tests.

Interesting.

So the script should list the remaining temporary directories after having
run the tests. There is no need to parse the log file of this directories.

Then I'd like to see a small change, which would be that :

1) we rely on a user-provided base directory instead of /tmp, such as
${TMPDIR:-/tmp} since TMPDIR is very commonly defined ;

2) we create our own directory there using mkstemp and we display this
name when starting the tests ;

3) everything is created under this temporary directory, probably using
the test's basename, so that we don't have a mixture of old and new
tests at the same place, instead we have a single directory per run ;

4) we continue to delete successful entries from these directories,
and we can rmdir -p the temporary directory so that it completely
disappears if everything was successful.

It will then make it very convenient to figure what tests systematically
fail and compare the errors between runs.

We cannot rely on the fact there is no bug in varnishtest logging code ;)

Post by PiBa-NL
If you could give it another look i would appreciate that. Are there any
things that need to be added/changed about it before considering to add
it to haproxy sources branch?

Yes a few details.
It seems there are remaining useless piped sed processes (sed | sed) or
perhaps I have missed something.
There is no need to use awk to remove sections of lines. Have a look to cut
command which is more lightweight.

It depends if there are multiple spaces or not.

tr command accepts escaped characters as '\n' for new line character which
is more readable than '
' ;)

Be careful, "\n" is a gnu-ism or something like this. The only portable
way to use \n is to use \012.

For me with these remaining modifications/cleanup this script could be added
to haproxy sources.

And please make the varnishtest program configurable as is done in the
makefile. This will help those of us who installed it in a location not
present in the path, and will also help support multiple versions during
development.

Thanks again for this good work Pieter.

Seconded!

Willy

[PiBa-NL]

Hi Frederic, Willy,

Hello Pieter,

Do you intend to finalize this script? We would like to use it in
haproxy sources.
Note that varnishtest already uses TMPDIR variable in place of /tmp if
it is set in the environment.
Thanks again.
Fred.

Thanks for your advices and comments, to be honest i haven't looked at
the script for several days, got distracted by other things ;). So sorry
for late reply.

Just cleaned it up a bit. I guess its ready for another review.

I still have a '
' newline, with the IFS= but the \n and \012 didnt seem to work there..

I've tried to incorporate all suggestions. Lemme know if/what i missed :)

Regards,

PiBa-NL (Pieter)

[PiBa-NL]

Hi Frederic,

Post by PiBa-NL
I still have a '
' newline, with the IFS= but the \n and \012 didnt seem to work there..

Strangely on my PC with both bash and dash I do not have to change
IFS value to parse HAPROXY_VERSION, TARGET and OPTIONS with "read"
internal command.

Reading version,target and options works fine indeed without, however
the loop over test files fails if any .vtc file has a space character in
its filename. Or should we 'forbid' that with documentation.? Or is
there another better workaround for that?

   TESTRUNDATETIME="$(date '+%Y-%m-%d_%H-%M-%S')"
   TESTDIR=${TMPDIR:-/tmp}/varnishtest_haproxy/$TESTRUNDATETIME
   mkdir -p $TESTDIR
What if we run the tests several times at the same time?

Well they would have to run at the same second. Not sure if that would
be wise to do.. But mktemp now solved that i guess :) at least for the
directory name part..

Please have a look to mkstemp utility.

Without the 's' right? Done, combined with the rundatetime which i do
like, so its 'readable' and unique, best of both ways i think?.

    cat $i/LOG | grep -- ----
should be replaced by
    grep -- ---- $i/LOG

I guess ill never learn do do this right the first time around ;). Fixed.

Note that you script is plenty of '\r' characters with no newline

Not sure what the 'correct' way would be. I think there is a CR LF
everywhere at the moment? And the scripts hashbang tries to point to
'sh', will this be a issue.? (acme.sh does the same, and seems to be run
an lots of systems..) And if so what can i best do to avoid issues?
Used a single = now.

When I do not set both HAPROY_PROGRAM I get this output with a script
with a successful result.

Checks added to avoid this issue for both haproxy and varnishtest so
they are checked to exist.

Next round :).

Regards,

PiBa-NL (Pieter)