path: root/build2/test/testscript

- Test description language (testscript) [feature]

Requirements
============

* Run multiple tests

* Multiple testscript files (to group related test)

* Specify exit status (optional, expect 0 by default).

* Run pre/post test actions (setups/teardowns, e.g., database schema creation).

* Test input (optional)

* Test stdout output (to compare to, optional)

* Test stderr output (to compare to, optional)

* Test description

* Create (and automatically clean up) input/output files

? Variable expansion, looked up on target in buildfile

* Ability to pass arguments from build files (how for multiple scripts?)
  Could use $0, $1, $*.

* Ability to pass arguments from command line.

? if/else, loops (we will have ternary operator as part of eval context)

* Multi-line comments (to disable a chunk of test; #\)

* Ability to suppress output (/dev/null)

* Ability not to clean up output (if failed)

* This should be consistent/uniform with how we handle simple tests without
  testscripts (i.e., perhaps they can be treated "as if" they had a one-line
  testscript).

Questions
=========

* What if the testcript attached at the buildfile level? An executable? What
  if the test is not for an executable. I guess we always have the directory
  target.

  We could reuse the test variable to specify (list of) scripts:

  exe{hello}: test = test1 test2

  Or maybe testscript should be a target (usually in src) itself?

* Do we want a notion of multi-command test (e.g., with pre/post commands)?
  Maybe {}? Will need it not to clean up output files too early. Also var
  scope.

* What if we need to define some global variables for the entire script?

* Ability to include testscripts (for grouping/organization purposes). Also
  include common definitions?

* Ability to redefine $0 (e.g., to src_base to run scripts)?

* What extension to use for script files? Should it be conventional or
  configurable/enforced?

  .t, .test, .ts, .tst, .bt, <none>

Notes
=====

* If expected exist status is not 0, then probably makes sense to suppress
  stderr output if there is no comparison.

Syntax
======

The goal is to be as concise as possible at least for the common cases.

$0 World       # hello World, 0 exit expected
$0 "John Doe"  # hello "John Doe"
$0 == 1        # hello, 1 exit expected
$0 != 0        # hello, non-0 exit expected

$0 World >>EOO
Hello, World!
EOO

$0 World >"Hello, World!"

$0 2>>EOE
Usage: $0 <name>
EOE

$0 --trace <<EOI >>EOO 2>>EOE # The data should appear in the order specified.
World
EOI
Hello, World!
EOO
13 characters written
EOE

~ Built-in cat command with auto-cleanup? To create files for tests, etc.

~ Built-in sed(-like) utility? Note that GCC 4.8 has broken <regex> support,
  only usable from 4.9. Could be a problem.

~ We should probably allow assignment of variables and since we will also
  lookup those defined in the buildfile, makes sense to use the same semantics
  (and perhaps even the parser, like we do for command line). Also value
  types.

  So in a sense for each test we will have an "inner" variable scope where
  we lookup first.

~ Could support assigning output of a program to a variable:

  foo = `sed ...`

  The semantics should probably be "as if you had the output literally",
  without any of the extra shell splitting nonsense, just the standard
  buildfile logic.

  Probably also support piping.

~ Output file redirects should be relative to out_base. Input? Could actually
  track created out files and use that if exists, and src otherwise. What if
  the test create a file without a redirect? Will need to "register" it
  somehow for auto-cleanup. Maybe &<file>? Also for directories &<dir>/.

  Does this mean that any program argument starting with & is recognized to
  be a file/directory for auto-cleanup?

~ Will there be a way to combine stderr and stdout, could be useful sometimes
  (e.g., order of output relative to diagnostics). 2>& syntax?

~ Do we support command piping?

~ What if we actually do want the output to go to a file, e.g., for a second
  command to act on it? Maybe keep shell syntax since familiar?  Or reverse it
  (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I
  like it. What about appending to file - >>>>? Maybe >&file?

~ $0 is target path (works out in case of a directory, but will be out).

~ #-comment (also for mid-line)

~ quoting & escaping (need to think and specify). Would be nice to avoid shell
  madness. Perhaps only support escaping only inside quoted strings).

~ If we support variable expansion inside "here doc", what if we need to
  specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only
  do effective escaping? Otherwise will have to escape backslashes...

~ Will probably need to support both $foo and $(foo) syntax. Will need to
  replicate var name lexing mode form build2. Perhaps use it to avoid
  duplication?

~ line continuation with \

~ == !=

~ Specify named variables instead/in addition to arguments

  ...: test = test1 test2
  ...: test.test1.var1 = ...

~ Perhaps the default variable type should be strings, not names? Thought
  we reverse most things pretty well.

~ CWD should probably be running tests in out_base (so if they create some
  files).

~ Clean up is tricky: sometimes we may want to keep it, but then how to clean
  it up later (the test might fail if there is junk present). Maybe create
  temp directory in out_base and run there? Will also help with parallel runs.

~ We will need an ability to execute same command multiple times with some
  argument variations. For simple cases the following idiom can be used:

  cmd = a b c
  $cmd d e f     # Expands to: a b c d e f
  v=`$cmd x y z`

  The problem is that only the last arguments (of the last program in pipe) can
  be varied.

  What if we can support variable templates with the following syntax:

  # Template variable definition. Same as a regular variable assignement but
  # $n variables are not expanded and denotes template parameters.
  #
  cmd ~= a $1 b | c $2

  # Instantiations. First n values that follows template variable expansion are
  # template arguments.
  #
  $cmd d e f   # Expands to: a d b | c e f
  v=`$cmd x y`

  Is this really common, this need to have a piped multi-command and to pass
  arguments to more than one of them? Remember, the idea is that tests must
  be as direct (literal) as possible. Any kind of indirection (or hiding
  things in variables) would just make stuff harder to understand. So maybe
  something like this should be just written as:

  a d b | c e f
  v=`a x b | c y f`

  But would be nice to see a real use-case that needed this.

~ How to control script execution from the command line? What about the
  followig approach.

  For test operation build2 assumes arguments which follows a special marker
  (-) to be intended for a test script target and get assigned to a special
  variable ($@). Test script normally just pass them to the program being
  tested. In the absense of the marker all unrecognized arguments are assigned
  to $@. So effectivelly the marker is required only to disambiguate build2 own
  arguments from a test target ones. Test script do not extract any information
  from $@ (while can modify it as any other variable).

  To ammend a test script execution the command line variables can be used.
  Probably it make sense to require such a variable to be defined in a
  buildfile to make sure it is initialized (with a default value). Example:

    buildfile:

      ./: test = test1
      ./: test.test1.c = ""
      ./: test.test1.remote = false
      ./: test.test1.verbose = false

    test1 script:

      $@ = (verbose ? : "-q") + $@
      cxx_options = (c != "" ? "config.cxx=$c" : )

    Command line:

      $ b test remote=true c=clang++ verbose=true - --verbose 4

  To define variable for all target test script we could use a special script
  name (*):

    ./: test = test1 test2
    ./: test.*.c = ""

  It is also tempting to extract test script variable values from command line
  options, so the above example could be changed in the following way:

    buildfile:

      ./: test = test1
      ./: test.test1.c = ""
      ./: test.test1.remote = false
      ./: test.test1.verbose = 0

    test1 script:

      $@ = (verbose == 0 ? "-q" : ) + $@
      cxx_options = (c != "" ? "config.cxx=$c" : )

    Command line:

      $ b test remote=true c=clang++ - --verbose 4