+ Test description language (testscript) [feature] Requirements ============ + Run multiple tests + Multiple testscript files (to group related test) + Specify exit status (optional, expect 0 by default). + Run pre/post test actions (setups/teardowns, e.g., database schema creation). + Test input (optional) + Test stdout output (to compare to, optional) + Test stderr output (to compare to, optional) + Test description + Create (and automatically clean up) input/output files + Variable expansion, looked up on target in buildfile + Ability to pass arguments from build files (how for multiple scripts?) Could use $0, $1, $*. * Ability to pass arguments from command line. + if/else, loops (we will have ternary operator as part of eval context) + Multi-line comments (to disable a chunk of test; #\) + Ability to suppress output (/dev/null) + Ability not to clean up output (if failed) * This should be consistent/uniform with how we handle simple tests without testscripts (i.e., perhaps they can be treated "as if" they had a one-line testscript). Questions ========= * What if the testcript attached at the buildfile level? An executable? What if the test is not for an executable. I guess we always have the directory target. We could reuse the test variable to specify (list of) scripts: exe{hello}: test = test1 test2 Or maybe testscript should be a target (usually in src) itself? * Do we want a notion of multi-command test (e.g., with pre/post commands)? Maybe {}? Will need it not to clean up output files too early. Also var scope. * What if we need to define some global variables for the entire script? * Ability to include testscripts (for grouping/organization purposes). Also include common definitions? * Ability to redefine $0 (e.g., to src_base to run scripts)? * What extension to use for script files? Should it be conventional or configurable/enforced? .t, .test, .ts, .tst, .bt, .bts .testscript just testscript (but if multiple tests then could be desirable to name them appropriately). Maybe either 'testscript' or '.testscript'? Notes ===== * If expected exit status is not 0, then probably makes sense to suppress stderr output if there is no stderr comparison. No, could be unexpected. Syntax ====== The goal is to be as concise as possible at least for the common cases. $0 World # hello World, 0 exit expected $0 "John Doe" # hello "John Doe" $0 == 1 # hello, 1 exit expected $0 != 0 # hello, non-0 exit expected $0 World >>EOO Hello, World! EOO $0 World >"Hello, World!" $0 2>>EOE Usage: $0 EOE $0 --trace <>EOO 2>>EOE # The data should appear in the order specified. World EOI Hello, World! EOO 13 characters written EOE ~ Built-in cat command with auto-cleanup? To create files for tests, etc. ~ Built-in sed(-like) utility? Note that GCC 4.8 has broken support, only usable from 4.9. Could be a problem. ~ We should probably allow assignment of variables and since we will also lookup those defined in the buildfile, makes sense to use the same semantics (and perhaps even the parser, like we do for command line). Also value types. So in a sense for each test we will have an "inner" variable scope where we lookup first. ~ Could support assigning output of a program to a variable: foo = `sed ...` The semantics should probably be "as if you had the output literally", without any of the extra shell splitting nonsense, just the standard buildfile logic. Probably also support piping. ~ Output file redirects should be relative to out_base. Input? Could actually track created out files and use that if exists, and src otherwise. What if the test create a file without a redirect? Will need to "register" it somehow for auto-cleanup. Maybe &? Also for directories &/. Does this mean that any program argument starting with & is recognized to be a file/directory for auto-cleanup? ~ Will there be a way to combine stderr and stdout, could be useful sometimes (e.g., order of output relative to diagnostics). 2>& syntax? ~ Do we support command piping? ~ What if we actually do want the output to go to a file, e.g., for a second command to act on it? Maybe keep shell syntax since familiar? Or reverse it (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I like it. What about appending to file - >>>>? Maybe >>>&file? ~ $0 is target path (works out in case of a directory, but will be out). ~ #-comment (also for mid-line) ~ quoting & escaping (need to think and specify). Would be nice to avoid shell madness. Perhaps only support escaping only inside quoted strings). ~ If we support variable expansion inside "here doc", what if we need to specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only do effective escaping? Otherwise will have to escape backslashes... ~ Will probably need to support both $foo and $(foo) syntax. Will need to replicate var name lexing mode form build2. Perhaps use it to avoid duplication? ~ line continuation with \ ~ == != ~ Specify named variables instead/in addition to arguments ...: test = test1 test2 ...: test.test1.var1 = ... ~ Perhaps the default variable type should be strings, not names? Thought we reverse most things pretty well. ~ CWD should probably be running tests in out_base (so if they create some files). ~ Clean up is tricky: sometimes we may want to keep it, but then how to clean it up later (the test might fail if there is junk present). Maybe create temp directory in out_base and run there? Will also help with parallel runs. ~ We will need an ability to execute same command multiple times with some argument variations. For simple cases the following idiom can be used: cmd = a b c $cmd d e f # Expands to: a b c d e f v=`$cmd x y z` The problem is that only the last arguments (of the last program in pipe) can be varied. What if we can support variable templates with the following syntax: # Template variable definition. Same as a regular variable assignement but # $n variables are not expanded and denotes template parameters. # cmd ~= a $1 b | c $2 # Instantiations. First n values that follows template variable expansion are # template arguments. # $cmd d e f # Expands to: a d b | c e f v=`$cmd x y` Is this really common, this need to have a piped multi-command and to pass arguments to more than one of them? Remember, the idea is that tests must be as direct (literal) as possible. Any kind of indirection (or hiding things in variables) would just make stuff harder to understand. So maybe something like this should be just written as: a d b | c e f v=`a x b | c y f` But would be nice to see a real use-case that needed this. ~ How to control script execution from the command line? What about the followig approach. For test operation build2 assumes arguments which follows a special marker (-) to be intended for a test script target and get assigned to a special variable ($@). Test script normally just pass them to the program being tested. In the absense of the marker all unrecognized arguments are assigned to $@. So effectivelly the marker is required only to disambiguate build2 own arguments from a test target ones. Test script do not extract any information from $@ (while can modify it as any other variable). To ammend a test script execution the command line variables can be used. Probably it make sense to require such a variable to be defined in a buildfile to make sure it is initialized (with a default value). Example: buildfile: ./: test = test1 ./: test.test1.c = "" ./: test.test1.remote = false ./: test.test1.verbose = false test1 script: $@ = (verbose ? : "-q") + $@ cxx_options = (c != "" ? "config.cxx=$c" : ) Command line: $ b test remote=true c=clang++ verbose=true - --verbose 4 To define variable for all target test script we could use a special script name (*): ./: test = test1 test2 ./: test.*.c = "" It is also tempting to extract test script variable values from command line options, so the above example could be changed in the following way: buildfile: ./: test = test1 ./: test.test1.c = "" ./: test.test1.remote = false ./: test.test1.verbose = 0 test1 script: $@ = (verbose == 0 ? "-q" : ) + $@ cxx_options = (c != "" ? "config.cxx=$c" : ) Command line: $ b test remote=true c=clang++ - --verbose 4