summaryrefslogtreecommitdiff
path: root/reference
diff options
context:
space:
mode:
authorBoris Kolpackov <boris@codesynthesis.com>2016-12-17 17:18:16 +0200
committerBoris Kolpackov <boris@codesynthesis.com>2016-12-17 17:18:16 +0200
commitdf70b9407714ea9846078c3f727977c501ad2349 (patch)
tree08cf741e2af06c69d38540eb823a3c5286efb16a /reference
parentcee59eb9d872acbbf973e8f8f9a05a8c9cb9fb44 (diff)
Implement test description language (testscript)
Diffstat (limited to 'reference')
-rw-r--r--reference/build2/test/testscript/testscript276
1 files changed, 276 insertions, 0 deletions
diff --git a/reference/build2/test/testscript/testscript b/reference/build2/test/testscript/testscript
new file mode 100644
index 0000000..1621389
--- /dev/null
+++ b/reference/build2/test/testscript/testscript
@@ -0,0 +1,276 @@
++ Test description language (testscript) [feature]
+
+Requirements
+============
+
++ Run multiple tests
+
++ Multiple testscript files (to group related test)
+
++ Specify exit status (optional, expect 0 by default).
+
++ Run pre/post test actions (setups/teardowns, e.g., database schema creation).
+
++ Test input (optional)
+
++ Test stdout output (to compare to, optional)
+
++ Test stderr output (to compare to, optional)
+
++ Test description
+
++ Create (and automatically clean up) input/output files
+
++ Variable expansion, looked up on target in buildfile
+
++ Ability to pass arguments from build files (how for multiple scripts?)
+ Could use $0, $1, $*.
+
+* Ability to pass arguments from command line.
+
++ if/else, loops (we will have ternary operator as part of eval context)
+
++ Multi-line comments (to disable a chunk of test; #\)
+
++ Ability to suppress output (/dev/null)
+
++ Ability not to clean up output (if failed)
+
+* This should be consistent/uniform with how we handle simple tests without
+ testscripts (i.e., perhaps they can be treated "as if" they had a one-line
+ testscript).
+
+Questions
+=========
+
+* What if the testcript attached at the buildfile level? An executable? What
+ if the test is not for an executable. I guess we always have the directory
+ target.
+
+ We could reuse the test variable to specify (list of) scripts:
+
+ exe{hello}: test = test1 test2
+
+ Or maybe testscript should be a target (usually in src) itself?
+
+* Do we want a notion of multi-command test (e.g., with pre/post commands)?
+ Maybe {}? Will need it not to clean up output files too early. Also var
+ scope.
+
+* What if we need to define some global variables for the entire script?
+
+* Ability to include testscripts (for grouping/organization purposes). Also
+ include common definitions?
+
+* Ability to redefine $0 (e.g., to src_base to run scripts)?
+
+* What extension to use for script files? Should it be conventional or
+ configurable/enforced?
+
+ .t, .test, .ts, .tst, .bt, <none>
+
+ .bts .testscript
+
+ just testscript (but if multiple tests then could be desirable to name them
+ appropriately).
+
+ Maybe either 'testscript' or '<name>.testscript'?
+
+Notes
+=====
+
+* If expected exit status is not 0, then probably makes sense to suppress
+ stderr output if there is no stderr comparison. No, could be unexpected.
+
+Syntax
+======
+
+The goal is to be as concise as possible at least for the common cases.
+
+$0 World # hello World, 0 exit expected
+$0 "John Doe" # hello "John Doe"
+$0 == 1 # hello, 1 exit expected
+$0 != 0 # hello, non-0 exit expected
+
+$0 World >>EOO
+Hello, World!
+EOO
+
+$0 World >"Hello, World!"
+
+$0 2>>EOE
+Usage: $0 <name>
+EOE
+
+$0 --trace <<EOI >>EOO 2>>EOE # The data should appear in the order specified.
+World
+EOI
+Hello, World!
+EOO
+13 characters written
+EOE
+
+~ Built-in cat command with auto-cleanup? To create files for tests, etc.
+
+~ Built-in sed(-like) utility? Note that GCC 4.8 has broken <regex> support,
+ only usable from 4.9. Could be a problem.
+
+~ We should probably allow assignment of variables and since we will also
+ lookup those defined in the buildfile, makes sense to use the same semantics
+ (and perhaps even the parser, like we do for command line). Also value
+ types.
+
+ So in a sense for each test we will have an "inner" variable scope where
+ we lookup first.
+
+~ Could support assigning output of a program to a variable:
+
+ foo = `sed ...`
+
+ The semantics should probably be "as if you had the output literally",
+ without any of the extra shell splitting nonsense, just the standard
+ buildfile logic.
+
+ Probably also support piping.
+
+~ Output file redirects should be relative to out_base. Input? Could actually
+ track created out files and use that if exists, and src otherwise. What if
+ the test create a file without a redirect? Will need to "register" it
+ somehow for auto-cleanup. Maybe &<file>? Also for directories &<dir>/.
+
+ Does this mean that any program argument starting with & is recognized to
+ be a file/directory for auto-cleanup?
+
+~ Will there be a way to combine stderr and stdout, could be useful sometimes
+ (e.g., order of output relative to diagnostics). 2>& syntax?
+
+~ Do we support command piping?
+
+~ What if we actually do want the output to go to a file, e.g., for a second
+ command to act on it? Maybe keep shell syntax since familiar? Or reverse it
+ (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I
+ like it. What about appending to file - >>>>? Maybe >>>&file?
+
+~ $0 is target path (works out in case of a directory, but will be out).
+
+~ #-comment (also for mid-line)
+
+~ quoting & escaping (need to think and specify). Would be nice to avoid shell
+ madness. Perhaps only support escaping only inside quoted strings).
+
+~ If we support variable expansion inside "here doc", what if we need to
+ specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only
+ do effective escaping? Otherwise will have to escape backslashes...
+
+~ Will probably need to support both $foo and $(foo) syntax. Will need to
+ replicate var name lexing mode form build2. Perhaps use it to avoid
+ duplication?
+
+~ line continuation with \
+
+~ == !=
+
+~ Specify named variables instead/in addition to arguments
+
+ ...: test = test1 test2
+ ...: test.test1.var1 = ...
+
+~ Perhaps the default variable type should be strings, not names? Thought
+ we reverse most things pretty well.
+
+~ CWD should probably be running tests in out_base (so if they create some
+ files).
+
+~ Clean up is tricky: sometimes we may want to keep it, but then how to clean
+ it up later (the test might fail if there is junk present). Maybe create
+ temp directory in out_base and run there? Will also help with parallel runs.
+
+~ We will need an ability to execute same command multiple times with some
+ argument variations. For simple cases the following idiom can be used:
+
+ cmd = a b c
+ $cmd d e f # Expands to: a b c d e f
+ v=`$cmd x y z`
+
+ The problem is that only the last arguments (of the last program in pipe) can
+ be varied.
+
+ What if we can support variable templates with the following syntax:
+
+ # Template variable definition. Same as a regular variable assignement but
+ # $n variables are not expanded and denotes template parameters.
+ #
+ cmd ~= a $1 b | c $2
+
+ # Instantiations. First n values that follows template variable expansion are
+ # template arguments.
+ #
+ $cmd d e f # Expands to: a d b | c e f
+ v=`$cmd x y`
+
+ Is this really common, this need to have a piped multi-command and to pass
+ arguments to more than one of them? Remember, the idea is that tests must
+ be as direct (literal) as possible. Any kind of indirection (or hiding
+ things in variables) would just make stuff harder to understand. So maybe
+ something like this should be just written as:
+
+ a d b | c e f
+ v=`a x b | c y f`
+
+ But would be nice to see a real use-case that needed this.
+
+~ How to control script execution from the command line? What about the
+ followig approach.
+
+ For test operation build2 assumes arguments which follows a special marker
+ (-) to be intended for a test script target and get assigned to a special
+ variable ($@). Test script normally just pass them to the program being
+ tested. In the absense of the marker all unrecognized arguments are assigned
+ to $@. So effectivelly the marker is required only to disambiguate build2 own
+ arguments from a test target ones. Test script do not extract any information
+ from $@ (while can modify it as any other variable).
+
+ To ammend a test script execution the command line variables can be used.
+ Probably it make sense to require such a variable to be defined in a
+ buildfile to make sure it is initialized (with a default value). Example:
+
+ buildfile:
+
+ ./: test = test1
+ ./: test.test1.c = ""
+ ./: test.test1.remote = false
+ ./: test.test1.verbose = false
+
+ test1 script:
+
+ $@ = (verbose ? : "-q") + $@
+ cxx_options = (c != "" ? "config.cxx=$c" : )
+
+ Command line:
+
+ $ b test remote=true c=clang++ verbose=true - --verbose 4
+
+ To define variable for all target test script we could use a special script
+ name (*):
+
+ ./: test = test1 test2
+ ./: test.*.c = ""
+
+ It is also tempting to extract test script variable values from command line
+ options, so the above example could be changed in the following way:
+
+ buildfile:
+
+ ./: test = test1
+ ./: test.test1.c = ""
+ ./: test.test1.remote = false
+ ./: test.test1.verbose = 0
+
+ test1 script:
+
+ $@ = (verbose == 0 ? "-q" : ) + $@
+ cxx_options = (c != "" ? "config.cxx=$c" : )
+
+ Command line:
+
+ $ b test remote=true c=clang++ - --verbose 4