Implement test description language (testscript)

1 files changed, 0 insertions, 276 deletions

diff --git a/build2/test/testscript/testscript b/build2/test/testscript/testscript
deleted file mode 100644
index b61280c..0000000
--- a/build2/test/testscript/testscript
+++ /dev/null
@@ -1,276 +0,0 @@
-- Test description language (testscript) [feature]
-
-Requirements
-============
-
-* Run multiple tests
-
-* Multiple testscript files (to group related test)
-
-* Specify exit status (optional, expect 0 by default).
-
-* Run pre/post test actions (setups/teardowns, e.g., database schema creation).
-
-* Test input (optional)
-
-* Test stdout output (to compare to, optional)
-
-* Test stderr output (to compare to, optional)
-
-* Test description
-
-* Create (and automatically clean up) input/output files
-
-? Variable expansion, looked up on target in buildfile
-
-* Ability to pass arguments from build files (how for multiple scripts?)
-  Could use $0, $1, $*.
-
-* Ability to pass arguments from command line.
-
-? if/else, loops (we will have ternary operator as part of eval context)
-
-* Multi-line comments (to disable a chunk of test; #\)
-
-* Ability to suppress output (/dev/null)
-
-* Ability not to clean up output (if failed)
-
-* This should be consistent/uniform with how we handle simple tests without
-  testscripts (i.e., perhaps they can be treated "as if" they had a one-line
-  testscript).
-
-Questions
-=========
-
-* What if the testcript attached at the buildfile level? An executable? What
-  if the test is not for an executable. I guess we always have the directory
-  target.
-
-  We could reuse the test variable to specify (list of) scripts:
-
-  exe{hello}: test = test1 test2
-
-  Or maybe testscript should be a target (usually in src) itself?
-
-* Do we want a notion of multi-command test (e.g., with pre/post commands)?
-  Maybe {}? Will need it not to clean up output files too early. Also var
-  scope.
-
-* What if we need to define some global variables for the entire script?
-
-* Ability to include testscripts (for grouping/organization purposes). Also
-  include common definitions?
-
-* Ability to redefine $0 (e.g., to src_base to run scripts)?
-
-* What extension to use for script files? Should it be conventional or
-  configurable/enforced?
-
-  .t, .test, .ts, .tst, .bt, <none>
-
-  .bts .testscript
-
-   just testscript (but if multiple tests then could be desirable to name them
-   appropriately).
-
-   Maybe either 'testscript' or '<name>.testscript'?
-
-Notes
-=====
-
-* If expected exit status is not 0, then probably makes sense to suppress
-  stderr output if there is no stderr comparison.
-
-Syntax
-======
-
-The goal is to be as concise as possible at least for the common cases.
-
-$0 World       # hello World, 0 exit expected
-$0 "John Doe"  # hello "John Doe"
-$0 == 1        # hello, 1 exit expected
-$0 != 0        # hello, non-0 exit expected
-
-$0 World >>EOO
-Hello, World!
-EOO
-
-$0 World >"Hello, World!"
-
-$0 2>>EOE
-Usage: $0 <name>
-EOE
-
-$0 --trace <<EOI >>EOO 2>>EOE # The data should appear in the order specified.
-World
-EOI
-Hello, World!
-EOO
-13 characters written
-EOE
-
-~ Built-in cat command with auto-cleanup? To create files for tests, etc.
-
-~ Built-in sed(-like) utility? Note that GCC 4.8 has broken <regex> support,
-  only usable from 4.9. Could be a problem.
-
-~ We should probably allow assignment of variables and since we will also
-  lookup those defined in the buildfile, makes sense to use the same semantics
-  (and perhaps even the parser, like we do for command line). Also value
-  types.
-
-  So in a sense for each test we will have an "inner" variable scope where
-  we lookup first.
-
-~ Could support assigning output of a program to a variable:
-
-  foo = `sed ...`
-
-  The semantics should probably be "as if you had the output literally",
-  without any of the extra shell splitting nonsense, just the standard
-  buildfile logic.
-
-  Probably also support piping.
-
-~ Output file redirects should be relative to out_base. Input? Could actually
-  track created out files and use that if exists, and src otherwise. What if
-  the test create a file without a redirect? Will need to "register" it
-  somehow for auto-cleanup. Maybe &<file>? Also for directories &<dir>/.
-
-  Does this mean that any program argument starting with & is recognized to
-  be a file/directory for auto-cleanup?
-
-~ Will there be a way to combine stderr and stdout, could be useful sometimes
-  (e.g., order of output relative to diagnostics). 2>& syntax?
-
-~ Do we support command piping?
-
-~ What if we actually do want the output to go to a file, e.g., for a second
-  command to act on it? Maybe keep shell syntax since familiar?  Or reverse it
-  (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I
-  like it. What about appending to file - >>>>? Maybe >>>&file?
-
-~ $0 is target path (works out in case of a directory, but will be out).
-
-~ #-comment (also for mid-line)
-
-~ quoting & escaping (need to think and specify). Would be nice to avoid shell
-  madness. Perhaps only support escaping only inside quoted strings).
-
-~ If we support variable expansion inside "here doc", what if we need to
-  specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only
-  do effective escaping? Otherwise will have to escape backslashes...
-
-~ Will probably need to support both $foo and $(foo) syntax. Will need to
-  replicate var name lexing mode form build2. Perhaps use it to avoid
-  duplication?
-
-~ line continuation with \
-
-~ == !=
-
-~ Specify named variables instead/in addition to arguments
-
-  ...: test = test1 test2
-  ...: test.test1.var1 = ...
-
-~ Perhaps the default variable type should be strings, not names? Thought
-  we reverse most things pretty well.
-
-~ CWD should probably be running tests in out_base (so if they create some
-  files).
-
-~ Clean up is tricky: sometimes we may want to keep it, but then how to clean
-  it up later (the test might fail if there is junk present). Maybe create
-  temp directory in out_base and run there? Will also help with parallel runs.
-
-~ We will need an ability to execute same command multiple times with some
-  argument variations. For simple cases the following idiom can be used:
-
-  cmd = a b c
-  $cmd d e f     # Expands to: a b c d e f
-  v=`$cmd x y z`
-
-  The problem is that only the last arguments (of the last program in pipe) can
-  be varied.
-
-  What if we can support variable templates with the following syntax:
-
-  # Template variable definition. Same as a regular variable assignement but
-  # $n variables are not expanded and denotes template parameters.
-  #
-  cmd ~= a $1 b | c $2
-
-  # Instantiations. First n values that follows template variable expansion are
-  # template arguments.
-  #
-  $cmd d e f   # Expands to: a d b | c e f
-  v=`$cmd x y`
-
-  Is this really common, this need to have a piped multi-command and to pass
-  arguments to more than one of them? Remember, the idea is that tests must
-  be as direct (literal) as possible. Any kind of indirection (or hiding
-  things in variables) would just make stuff harder to understand. So maybe
-  something like this should be just written as:
-
-  a d b | c e f
-  v=`a x b | c y f`
-
-  But would be nice to see a real use-case that needed this.
-
-~ How to control script execution from the command line? What about the
-  followig approach.
-
-  For test operation build2 assumes arguments which follows a special marker
-  (-) to be intended for a test script target and get assigned to a special
-  variable ($@). Test script normally just pass them to the program being
-  tested. In the absense of the marker all unrecognized arguments are assigned
-  to $@. So effectivelly the marker is required only to disambiguate build2 own
-  arguments from a test target ones. Test script do not extract any information
-  from $@ (while can modify it as any other variable).
-
-  To ammend a test script execution the command line variables can be used.
-  Probably it make sense to require such a variable to be defined in a
-  buildfile to make sure it is initialized (with a default value). Example:
-
-    buildfile:
-
-      ./: test = test1
-      ./: test.test1.c = ""
-      ./: test.test1.remote = false
-      ./: test.test1.verbose = false
-
-    test1 script:
-
-      $@ = (verbose ? : "-q") + $@
-      cxx_options = (c != "" ? "config.cxx=$c" : )
-
-    Command line:
-
-      $ b test remote=true c=clang++ verbose=true - --verbose 4
-
-  To define variable for all target test script we could use a special script
-  name (*):
-
-    ./: test = test1 test2
-    ./: test.*.c = ""
-
-  It is also tempting to extract test script variable values from command line
-  options, so the above example could be changed in the following way:
-
-    buildfile:
-
-      ./: test = test1
-      ./: test.test1.c = ""
-      ./: test.test1.remote = false
-      ./: test.test1.verbose = 0
-
-    test1 script:
-
-      $@ = (verbose == 0 ? "-q" : ) + $@
-      cxx_options = (c != "" ? "config.cxx=$c" : )
-
-    Command line:
-
-      $ b test remote=true c=clang++ - --verbose 4