Discussion:
[PATCH] REGTEST/MINOR: script: add run-regtests.sh script
PiBa-NL
2018-11-25 15:59:52 UTC
Permalink
Hi Frederic, Willy,

Added the varnishtest script we have been discussing as a .patch this time.

I put the script in the /reg-tests/ folder. Maybe it should have been
besides the Makefile in the / root ?

Also i put a bit of comments into the commit.

I hope it is okay like this? If not, feel free to comment on them or
change them as required.

Once this one is 'accepted' ill create a few patches for the existing
.vtc files to include their requirements. (at least the more obvious ones..)

Regards,
PiBa-NL (Pieter)
Willy Tarreau
2018-11-27 08:52:49 UTC
Permalink
Hi guys,
Post by PiBa-NL
I put the script in the /reg-tests/ folder. Maybe it should have been
besides the Makefile in the / root ?
Yes I think it should be placed at the same level as the Makefile.
Well, we already have a "scripts" directory with the stuff used for
release and backport management. I think it perfectly has its place
there.
+ for i in $(find $TESTDIR/ -type d -name "vtc.*");
+ do
+ echo "###### $(cat $i/INFO) ######"
+ echo "## test results in: $i"
+ grep -- ---- $i/LOG
+
+ echo "###### $(cat $i/INFO) ######" >> $TESTDIR/failedtests.log
+ echo "## test results in: $i" >> $TESTDIR/failedtests.log
+ grep -- ---- $i/LOG >> $TESTDIR/failedtests.log
+ echo >> $TESTDIR/failedtests.log
+ done
cat <<- EOF | tee $TESDIR/failedtests.log
.
.
EOF
I have tested you script. For me it is OK. Good job!
Thank you a lot Pieter.
OK just let me know what to do with this, should I merge it as-is and
expect minor updates later, or do you or Pieter want to resend an
updated version ? I can adapt, let me know.

thanks!
Willy
PiBa-NL
2018-11-27 22:17:42 UTC
Permalink
Hi Frederic, Willy,
Post by PiBa-NL
Post by Willy Tarreau
Hi guys,
Post by PiBa-NL
I put the script in the /reg-tests/ folder. Maybe it should have been
besides the Makefile in the / root ?
Yes I think it should be placed at the same level as the Makefile.
Well, we already have a "scripts" directory with the stuff used for
release and backport management. I think it perfectly has its place
there.
/scripts/ sounds good.
Post by PiBa-NL
Post by Willy Tarreau
Note that the reg tests must be run from the Makefile with
"reg-tests" target and possibly other arguments/variables.
Willy recently added REG_TEST_FILES variable.
I've changed the the script to include the LEVEL parameter almost the
way the Makefile used it, changed behavior though so without the
parameter it it runs all tests.
Post by PiBa-NL
Post by Willy Tarreau
+  for i in $(find $TESTDIR/ -type d -name "vtc.*");
+  do
+    echo "###### $(cat $i/INFO) ######"
+    echo "## test results in: $i"
+    grep -- ---- $i/LOG
+
+    echo "###### $(cat $i/INFO) ######" >> $TESTDIR/failedtests.log
+    echo "## test results in: $i" >> $TESTDIR/failedtests.log
+    grep -- ---- $i/LOG >> $TESTDIR/failedtests.log
+    echo >> $TESTDIR/failedtests.log
+  done
     cat <<- EOF | tee $TESDIR/failedtests.log
     .
     .
     EOF
Removed some spaces for indentation which became part of the output.
Post by PiBa-NL
Post by Willy Tarreau
I have tested you script. For me it is OK. Good job!
Thank you a lot Pieter.
OK just let me know what to do with this, should I merge it as-is and
expect minor updates later, or do you or Pieter want to resend an
updated version ? I can adapt, let me know.
I have modified Pieter's patch for the modification mentioned above.
Seems to work ;)
Willy,
Here is a better patch which takes into an account the modification
above and yours (the script is added in "tests" directory).
I think Willy mentioned a 'scripts' directory? Changed patch to include
that as well.
Post by PiBa-NL
You can merge it as-is.
Regards,
Fred
New path attached, which includes a LEVEL check.
And a modification of the Makefile to call the ./scripts/run-regtests.sh

Please can someone check it again before merging.?. Thanks guys :).

Regards,
PiBa-NL (Pieter)
Willy Tarreau
2018-11-28 03:48:45 UTC
Permalink
Hi Pieter,
I've changed the the script to include the LEVEL parameter almost the way
the Makefile used it, changed behavior though so without the parameter it it
runs all tests.
(...)
New path attached, which includes a LEVEL check.
And a modification of the Makefile to call the ./scripts/run-regtests.sh
Thank you.
Please can someone check it again before merging.?. Thanks guys :).
I've run a quick check and am mostly OK with it, though I'll wait for
Fred to re-check this morning before applying it.
diff --git a/Makefile b/Makefile
(...)
+ ./scripts/run-regtests.sh --LEVEL "$$LEVEL" $$REG_TEST_FILES
I'll reuse $(REG_TEST_FILES) instead of $$REG_TEST_FILES. The
difference is that the former is resolved by make and will take care
of makefile variables, while the latter is resolved by the shell and
will only see exported environment variables. This is fairly minor,
no need to respin a patch for this ;-)

Thanks,
Willy
Willy Tarreau
2018-11-29 04:36:35 UTC
Permalink
Hi guys,
Perhaps we should "chmod +x" this script.
Good point, done here.

However I'm now seeing this when starting it :

########################## Starting varnishtest ##########################
Testing with haproxy version: 1.8-dev0-3b0a6d-66
Assert error in start_test(), vtc_main.c line 283:
Condition((mkdir(tmpdir, 0711)) == 0) not true.
errno = 13 (Permission denied)
./scripts/run-regtests.sh: line 299: 21484 Aborted $VARNISHTEST_PROGRAM $varnishtestparams $verbose $jobcount -l -k -t 10 $testlist
########################## Gathering failed results ##########################
find: `/proc/tty/driver': Permission denied
find: `/proc/1/task/1/fd': Permission denied
find: `/proc/1/task/1/fdinfo': Permission denied
find: `/proc/1/task/1/ns': Permission denied
find: `/proc/1/fd': Permission denied
find: `/proc/1/map_files': Permission denied
(...)

Then I stopped it using Ctrl-C.

I'm seeing a few minor issues we must still address :

- haproxy is started from the path, which means that on all systems
where it's installed, it will be the one provided by the system and
not the just built one which will be tested (as happened above),
which is confusing. I think we shouldn't search for haproxy in the
path but use $PWD/haproxy or ./haproxy or something like this.

- it seems there are some issues in the way TMPDIR/TESTDIR are created,
as it ended up with an empty TESTDIR. In my case, TMPDIR is not set,
so TESTDIR was set to /tmp/varnishtest_haproxy. mkdir worked (I now
have this directory), but a test is missing on this mkdir.

- the way the sub-directory is created is problematic on shared
development machines, as only the first user will own the directory
and will thus prevent other users from creating their own overthere,
thus it's probably preferable not to create an intermediary directory
in the end.

- in my case it's mktemp which failed :

++ date +%Y-%m-%d_%H-%M-%S
+ TESTRUNDATETIME=2018-11-29_05-03-43
+ TESTDIR=/tmp/varnishtest_haproxy
+ mkdir -p /tmp/varnishtest_haproxy
++ mktemp -d /tmp/varnishtest_haproxy/2018-11-29_05-03-43.XXXX
mktemp: cannot make temp dir /tmp/varnishtest_haproxy/2018-11-29_05-03-43.XXXX: Invalid argument
+ TESTDIR=

I haven't checked why yet, but we definitely need to test the status
code for success as well.

- in my opinion the script is still missing a number of quotes when using
string variables, especially in the directory names. If someone is crazy
enough to have TMPDIR="/tmp/Temp Dir", then he'll have some fun.

- I'm seeing a linux-specific "rm -d" at the end, it's be better to
replace it with rmdir.

- there's "/usr/bin/env sh" at the top of the shell script. /bin/sh is
the portable one, I've spent lots of time in the past editing scripts
where env was forced to /usr/bin while it was placed in /bin on some
systems, so I'm pretty certain that this one is not portable at all :-)

However I'm well aware that it's easier to work on improvements once the
script is merged, so what I've done now is to merge it and create a
temporary "reg-tests2" target in the makefile to use it without losing
the existing one. This way everyone can work in parallel, and once the
few issues seem reliably addressed, we can definitely replace the make
target.

Thanks!
Willy
Willy Tarreau
2018-11-29 07:47:53 UTC
Permalink
Post by Willy Tarreau
However I'm well aware that it's easier to work on improvements once the
script is merged, so what I've done now is to merge it and create a
temporary "reg-tests2" target in the makefile to use it without losing
the existing one. This way everyone can work in parallel, and once the
few issues seem reliably addressed, we can definitely replace the make
target.
Unfortunately ENOCOFFEE struck me this morning and I forgot to commit
my local changes so I merged the unmodified version which replaces the
"reg-test" target.

Thus now we're condemned to quickly fix these small issues :-)

I've found the problem related to mktemp, it uses "XXXX" as the suffix.
On my distro the man page says :

The mktemp utility takes the given filename template and overwrites a
portion of it to create a unique filename. The template may be any
filename with six (6) `Xs' appended to it, for example
/tmp/tfile.XXXXXX.

I've found that other systems demand at least 3 or at least 1. And the
glibc's mktemp() calll wants 6 anyway or returns EINVAL (the one I got).
Thus we should add 2 more "X" to make this work everywhere mktemp is
present (and test for the output as well).

Willy
PiBa-NL
2018-11-29 20:56:49 UTC
Permalink
Hi Frederic,
Post by Willy Tarreau
Post by Willy Tarreau
Post by Willy Tarreau
However I'm well aware that it's easier to work on improvements once the
script is merged, so what I've done now is to merge it and create a
temporary "reg-tests2" target in the makefile to use it without losing
the existing one. This way everyone can work in parallel, and once the
few issues seem reliably addressed, we can definitely replace the make
target.
Unfortunately ENOCOFFEE struck me this morning and I forgot to commit
my local changes so I merged the unmodified version which replaces the
"reg-test" target.
Thus now we're condemned to quickly fix these small issues :-)
Pieter,
I am having a look at all these issues.
Regards,
Fred.
If that means i don't have to do anything at this moment, thank you! (i
suppose your turnaround time from issue>fix will also be shorter than
waiting for my spare evening hour..)
Ill start checking some requirements of existing .vtc . And perhaps
writing new one's.

Regards,

PiBa-NL (Pieter)
Willy Tarreau
2018-11-30 04:59:31 UTC
Permalink
Here is a patch for haproxy (named 0001-REGTEST*) to fix these issues.
Works like a charm, thank you Fred! Now applied.

Willy

Loading...