planned test.sh improvments

[ct]

After some discussion on IRC I introduce some needed/planned improvements for tests.sh here. Implementing these needs moderate bash skills, anyone who want to help the project may try it, otherwise I'll do it when I have some spare time for it.

Refactor and add more flags for output specifiers:

Currently we have hardcoded output specifiers out, out-lit, err and err-lit. This should be rewritten so that the first word gives the stdio fdname (out or err) and any following word delimited by - acts as a flag. Then some shell variables should provide defaults and overrideable in tests.conf and testsuites themself. Defaults should stay as it currently is (*-rx-trig) for backwards compatibility.

Following flags have to be supported:

• -lit the following specification string has to be taken literally
• -rx the following specification string is a regex

Note that these are mutally exclusive and either one has to be provided (by defaults)

• -trig triggering behaviour, after one fail the next regex is tried, if that fails again then the test fails.
• -once the specification must match exactly one time

This 2 are mutally exclusive too same as above.

• -not inverses the match

(do we need any flags for the 'in' fd? be prepared at least)

Diverting output based on regex matches:

Currently test.sh expects outputs in very well defined order, this is a little problem when testing asynchronous multithreaded programs, so we introduce a new syntax to split output specifications into seraveral 'channels'. This is based on a parentized regex just before the delimiting ': ', the old syntax without this parens then takes any outputs not diverted into this channels. I expect that implemeneting this feature is a bit challenging.

Some examples:

out-rx(thread_1:): Matches this on any line also containing thread_1:
err-not: error

Include some stdout context into the testlog in case of an fail

Currently only stderr is included in testlog (completely) but data on stdout is left out. In case of an fail we want some last lines from stdout as well, since stdout is expected to be volumious and usually relatively boring this should be limited to some context around the detected error (+/-10 lines)

Better explanation and reports

Fails don't produce much insight on why they failed yet, we should keep a log with more details (what actually matched and what failed) and include the last few things in the testlog in case of an failure.

Hashing tests

currently tests are solely identified by their textual descriptor. This is used by the FAST and FIRSTFAIL testmodes. This is error prone in case of non-unique descriptions. Better hash the whole test-call and stuff that into a logfile which is then grepped for previous test results.

better abort on timeout

There are some glitches on the currently quite hacked code aborting a test after a timeout, figure that out and fix it.