Implement test description language (testscript)

1 files changed, 276 insertions, 0 deletions

diff --git a/reference/build2/test/testscript/testscript b/reference/build2/test/testscript/testscript
new file mode 100644
index 0000000..1621389
--- /dev/null
+++ b/reference/build2/test/testscript/testscript
@@ -0,0 +1,276 @@
++ Test description language (testscript) [feature]
+
+Requirements
+============
+
++ Run multiple tests
+
++ Multiple testscript files (to group related test)
+
++ Specify exit status (optional, expect 0 by default).
+
++ Run pre/post test actions (setups/teardowns, e.g., database schema creation).
+
++ Test input (optional)
+
++ Test stdout output (to compare to, optional)
+
++ Test stderr output (to compare to, optional)
+
++ Test description
+
++ Create (and automatically clean up) input/output files
+
++ Variable expansion, looked up on target in buildfile
+
++ Ability to pass arguments from build files (how for multiple scripts?)
+  Could use $0, $1, $*.
+
+* Ability to pass arguments from command line.
+
++ if/else, loops (we will have ternary operator as part of eval context)
+
++ Multi-line comments (to disable a chunk of test; #\)
+
++ Ability to suppress output (/dev/null)
+
++ Ability not to clean up output (if failed)
+
+* This should be consistent/uniform with how we handle simple tests without
+  testscripts (i.e., perhaps they can be treated "as if" they had a one-line
+  testscript).
+
+Questions
+=========
+
+* What if the testcript attached at the buildfile level? An executable? What
+  if the test is not for an executable. I guess we always have the directory
+  target.
+
+  We could reuse the test variable to specify (list of) scripts:
+
+  exe{hello}: test = test1 test2
+
+  Or maybe testscript should be a target (usually in src) itself?
+
+* Do we want a notion of multi-command test (e.g., with pre/post commands)?
+  Maybe {}? Will need it not to clean up output files too early. Also var
+  scope.
+
+* What if we need to define some global variables for the entire script?
+
+* Ability to include testscripts (for grouping/organization purposes). Also
+  include common definitions?
+
+* Ability to redefine $0 (e.g., to src_base to run scripts)?
+
+* What extension to use for script files? Should it be conventional or
+  configurable/enforced?
+
+  .t, .test, .ts, .tst, .bt, <none>
+
+  .bts .testscript
+
+   just testscript (but if multiple tests then could be desirable to name them
+   appropriately).
+
+   Maybe either 'testscript' or '<name>.testscript'?
+
+Notes
+=====
+
+* If expected exit status is not 0, then probably makes sense to suppress
+  stderr output if there is no stderr comparison. No, could be unexpected.
+
+Syntax
+======
+
+The goal is to be as concise as possible at least for the common cases.
+
+$0 World       # hello World, 0 exit expected
+$0 "John Doe"  # hello "John Doe"
+$0 == 1        # hello, 1 exit expected
+$0 != 0        # hello, non-0 exit expected
+
+$0 World >>EOO
+Hello, World!
+EOO
+
+$0 World >"Hello, World!"
+
+$0 2>>EOE
+Usage: $0 <name>
+EOE
+
+$0 --trace <<EOI >>EOO 2>>EOE # The data should appear in the order specified.
+World
+EOI
+Hello, World!
+EOO
+13 characters written
+EOE
+
+~ Built-in cat command with auto-cleanup? To create files for tests, etc.
+
+~ Built-in sed(-like) utility? Note that GCC 4.8 has broken <regex> support,
+  only usable from 4.9. Could be a problem.
+
+~ We should probably allow assignment of variables and since we will also
+  lookup those defined in the buildfile, makes sense to use the same semantics
+  (and perhaps even the parser, like we do for command line). Also value
+  types.
+
+  So in a sense for each test we will have an "inner" variable scope where
+  we lookup first.
+
+~ Could support assigning output of a program to a variable:
+
+  foo = `sed ...`
+
+  The semantics should probably be "as if you had the output literally",
+  without any of the extra shell splitting nonsense, just the standard
+  buildfile logic.
+
+  Probably also support piping.
+
+~ Output file redirects should be relative to out_base. Input? Could actually
+  track created out files and use that if exists, and src otherwise. What if
+  the test create a file without a redirect? Will need to "register" it
+  somehow for auto-cleanup. Maybe &<file>? Also for directories &<dir>/.
+
+  Does this mean that any program argument starting with & is recognized to
+  be a file/directory for auto-cleanup?
+
+~ Will there be a way to combine stderr and stdout, could be useful sometimes
+  (e.g., order of output relative to diagnostics). 2>& syntax?
+
+~ Do we support command piping?
+
+~ What if we actually do want the output to go to a file, e.g., for a second
+  command to act on it? Maybe keep shell syntax since familiar?  Or reverse it
+  (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I
+  like it. What about appending to file - >>>>? Maybe >>>&file?
+
+~ $0 is target path (works out in case of a directory, but will be out).
+
+~ #-comment (also for mid-line)
+
+~ quoting & escaping (need to think and specify). Would be nice to avoid shell
+  madness. Perhaps only support escaping only inside quoted strings).
+
+~ If we support variable expansion inside "here doc", what if we need to
+  specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only
+  do effective escaping? Otherwise will have to escape backslashes...
+
+~ Will probably need to support both $foo and $(foo) syntax. Will need to
+  replicate var name lexing mode form build2. Perhaps use it to avoid
+  duplication?
+
+~ line continuation with \
+
+~ == !=
+
+~ Specify named variables instead/in addition to arguments
+
+  ...: test = test1 test2
+  ...: test.test1.var1 = ...
+
+~ Perhaps the default variable type should be strings, not names? Thought
+  we reverse most things pretty well.
+
+~ CWD should probably be running tests in out_base (so if they create some
+  files).
+
+~ Clean up is tricky: sometimes we may want to keep it, but then how to clean
+  it up later (the test might fail if there is junk present). Maybe create
+  temp directory in out_base and run there? Will also help with parallel runs.
+
+~ We will need an ability to execute same command multiple times with some
+  argument variations. For simple cases the following idiom can be used:
+
+  cmd = a b c
+  $cmd d e f     # Expands to: a b c d e f
+  v=`$cmd x y z`
+
+  The problem is that only the last arguments (of the last program in pipe) can
+  be varied.
+
+  What if we can support variable templates with the following syntax:
+
+  # Template variable definition. Same as a regular variable assignement but
+  # $n variables are not expanded and denotes template parameters.
+  #
+  cmd ~= a $1 b | c $2
+
+  # Instantiations. First n values that follows template variable expansion are
+  # template arguments.
+  #
+  $cmd d e f   # Expands to: a d b | c e f
+  v=`$cmd x y`
+
+  Is this really common, this need to have a piped multi-command and to pass
+  arguments to more than one of them? Remember, the idea is that tests must
+  be as direct (literal) as possible. Any kind of indirection (or hiding
+  things in variables) would just make stuff harder to understand. So maybe
+  something like this should be just written as:
+
+  a d b | c e f
+  v=`a x b | c y f`
+
+  But would be nice to see a real use-case that needed this.
+
+~ How to control script execution from the command line? What about the
+  followig approach.
+
+  For test operation build2 assumes arguments which follows a special marker
+  (-) to be intended for a test script target and get assigned to a special
+  variable ($@). Test script normally just pass them to the program being
+  tested. In the absense of the marker all unrecognized arguments are assigned
+  to $@. So effectivelly the marker is required only to disambiguate build2 own
+  arguments from a test target ones. Test script do not extract any information
+  from $@ (while can modify it as any other variable).
+
+  To ammend a test script execution the command line variables can be used.
+  Probably it make sense to require such a variable to be defined in a
+  buildfile to make sure it is initialized (with a default value). Example:
+
+    buildfile:
+
+      ./: test = test1
+      ./: test.test1.c = ""
+      ./: test.test1.remote = false
+      ./: test.test1.verbose = false
+
+    test1 script:
+
+      $@ = (verbose ? : "-q") + $@
+      cxx_options = (c != "" ? "config.cxx=$c" : )
+
+    Command line:
+
+      $ b test remote=true c=clang++ verbose=true - --verbose 4
+
+  To define variable for all target test script we could use a special script
+  name (*):
+
+    ./: test = test1 test2
+    ./: test.*.c = ""
+
+  It is also tempting to extract test script variable values from command line
+  options, so the above example could be changed in the following way:
+
+    buildfile:
+
+      ./: test = test1
+      ./: test.test1.c = ""
+      ./: test.test1.remote = false
+      ./: test.test1.verbose = 0
+
+    test1 script:
+
+      $@ = (verbose == 0 ? "-q" : ) + $@
+      cxx_options = (c != "" ? "config.cxx=$c" : )
+
+    Command line:
+
+      $ b test remote=true c=clang++ - --verbose 4