diff options
Diffstat (limited to 'reference/build2/test/testscript/testscript')
| -rw-r--r-- | reference/build2/test/testscript/testscript | 276 |
1 files changed, 276 insertions, 0 deletions
diff --git a/reference/build2/test/testscript/testscript b/reference/build2/test/testscript/testscript new file mode 100644 index 0000000..1621389 --- /dev/null +++ b/reference/build2/test/testscript/testscript @@ -0,0 +1,276 @@ ++ Test description language (testscript) [feature] + +Requirements +============ + ++ Run multiple tests + ++ Multiple testscript files (to group related test) + ++ Specify exit status (optional, expect 0 by default). + ++ Run pre/post test actions (setups/teardowns, e.g., database schema creation). + ++ Test input (optional) + ++ Test stdout output (to compare to, optional) + ++ Test stderr output (to compare to, optional) + ++ Test description + ++ Create (and automatically clean up) input/output files + ++ Variable expansion, looked up on target in buildfile + ++ Ability to pass arguments from build files (how for multiple scripts?) + Could use $0, $1, $*. + +* Ability to pass arguments from command line. + ++ if/else, loops (we will have ternary operator as part of eval context) + ++ Multi-line comments (to disable a chunk of test; #\) + ++ Ability to suppress output (/dev/null) + ++ Ability not to clean up output (if failed) + +* This should be consistent/uniform with how we handle simple tests without + testscripts (i.e., perhaps they can be treated "as if" they had a one-line + testscript). + +Questions +========= + +* What if the testcript attached at the buildfile level? An executable? What + if the test is not for an executable. I guess we always have the directory + target. + + We could reuse the test variable to specify (list of) scripts: + + exe{hello}: test = test1 test2 + + Or maybe testscript should be a target (usually in src) itself? + +* Do we want a notion of multi-command test (e.g., with pre/post commands)? + Maybe {}? Will need it not to clean up output files too early. Also var + scope. + +* What if we need to define some global variables for the entire script? + +* Ability to include testscripts (for grouping/organization purposes). Also + include common definitions? + +* Ability to redefine $0 (e.g., to src_base to run scripts)? + +* What extension to use for script files? Should it be conventional or + configurable/enforced? + + .t, .test, .ts, .tst, .bt, <none> + + .bts .testscript + + just testscript (but if multiple tests then could be desirable to name them + appropriately). + + Maybe either 'testscript' or '<name>.testscript'? + +Notes +===== + +* If expected exit status is not 0, then probably makes sense to suppress + stderr output if there is no stderr comparison. No, could be unexpected. + +Syntax +====== + +The goal is to be as concise as possible at least for the common cases. + +$0 World # hello World, 0 exit expected +$0 "John Doe" # hello "John Doe" +$0 == 1 # hello, 1 exit expected +$0 != 0 # hello, non-0 exit expected + +$0 World >>EOO +Hello, World! +EOO + +$0 World >"Hello, World!" + +$0 2>>EOE +Usage: $0 <name> +EOE + +$0 --trace <<EOI >>EOO 2>>EOE # The data should appear in the order specified. +World +EOI +Hello, World! +EOO +13 characters written +EOE + +~ Built-in cat command with auto-cleanup? To create files for tests, etc. + +~ Built-in sed(-like) utility? Note that GCC 4.8 has broken <regex> support, + only usable from 4.9. Could be a problem. + +~ We should probably allow assignment of variables and since we will also + lookup those defined in the buildfile, makes sense to use the same semantics + (and perhaps even the parser, like we do for command line). Also value + types. + + So in a sense for each test we will have an "inner" variable scope where + we lookup first. + +~ Could support assigning output of a program to a variable: + + foo = `sed ...` + + The semantics should probably be "as if you had the output literally", + without any of the extra shell splitting nonsense, just the standard + buildfile logic. + + Probably also support piping. + +~ Output file redirects should be relative to out_base. Input? Could actually + track created out files and use that if exists, and src otherwise. What if + the test create a file without a redirect? Will need to "register" it + somehow for auto-cleanup. Maybe &<file>? Also for directories &<dir>/. + + Does this mean that any program argument starting with & is recognized to + be a file/directory for auto-cleanup? + +~ Will there be a way to combine stderr and stdout, could be useful sometimes + (e.g., order of output relative to diagnostics). 2>& syntax? + +~ Do we support command piping? + +~ What if we actually do want the output to go to a file, e.g., for a second + command to act on it? Maybe keep shell syntax since familiar? Or reverse it + (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I + like it. What about appending to file - >>>>? Maybe >>>&file? + +~ $0 is target path (works out in case of a directory, but will be out). + +~ #-comment (also for mid-line) + +~ quoting & escaping (need to think and specify). Would be nice to avoid shell + madness. Perhaps only support escaping only inside quoted strings). + +~ If we support variable expansion inside "here doc", what if we need to + specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only + do effective escaping? Otherwise will have to escape backslashes... + +~ Will probably need to support both $foo and $(foo) syntax. Will need to + replicate var name lexing mode form build2. Perhaps use it to avoid + duplication? + +~ line continuation with \ + +~ == != + +~ Specify named variables instead/in addition to arguments + + ...: test = test1 test2 + ...: test.test1.var1 = ... + +~ Perhaps the default variable type should be strings, not names? Thought + we reverse most things pretty well. + +~ CWD should probably be running tests in out_base (so if they create some + files). + +~ Clean up is tricky: sometimes we may want to keep it, but then how to clean + it up later (the test might fail if there is junk present). Maybe create + temp directory in out_base and run there? Will also help with parallel runs. + +~ We will need an ability to execute same command multiple times with some + argument variations. For simple cases the following idiom can be used: + + cmd = a b c + $cmd d e f # Expands to: a b c d e f + v=`$cmd x y z` + + The problem is that only the last arguments (of the last program in pipe) can + be varied. + + What if we can support variable templates with the following syntax: + + # Template variable definition. Same as a regular variable assignement but + # $n variables are not expanded and denotes template parameters. + # + cmd ~= a $1 b | c $2 + + # Instantiations. First n values that follows template variable expansion are + # template arguments. + # + $cmd d e f # Expands to: a d b | c e f + v=`$cmd x y` + + Is this really common, this need to have a piped multi-command and to pass + arguments to more than one of them? Remember, the idea is that tests must + be as direct (literal) as possible. Any kind of indirection (or hiding + things in variables) would just make stuff harder to understand. So maybe + something like this should be just written as: + + a d b | c e f + v=`a x b | c y f` + + But would be nice to see a real use-case that needed this. + +~ How to control script execution from the command line? What about the + followig approach. + + For test operation build2 assumes arguments which follows a special marker + (-) to be intended for a test script target and get assigned to a special + variable ($@). Test script normally just pass them to the program being + tested. In the absense of the marker all unrecognized arguments are assigned + to $@. So effectivelly the marker is required only to disambiguate build2 own + arguments from a test target ones. Test script do not extract any information + from $@ (while can modify it as any other variable). + + To ammend a test script execution the command line variables can be used. + Probably it make sense to require such a variable to be defined in a + buildfile to make sure it is initialized (with a default value). Example: + + buildfile: + + ./: test = test1 + ./: test.test1.c = "" + ./: test.test1.remote = false + ./: test.test1.verbose = false + + test1 script: + + $@ = (verbose ? : "-q") + $@ + cxx_options = (c != "" ? "config.cxx=$c" : ) + + Command line: + + $ b test remote=true c=clang++ verbose=true - --verbose 4 + + To define variable for all target test script we could use a special script + name (*): + + ./: test = test1 test2 + ./: test.*.c = "" + + It is also tempting to extract test script variable values from command line + options, so the above example could be changed in the following way: + + buildfile: + + ./: test = test1 + ./: test.test1.c = "" + ./: test.test1.remote = false + ./: test.test1.verbose = 0 + + test1 script: + + $@ = (verbose == 0 ? "-q" : ) + $@ + cxx_options = (c != "" ? "config.cxx=$c" : ) + + Command line: + + $ b test remote=true c=clang++ - --verbose 4 |