From df70b9407714ea9846078c3f727977c501ad2349 Mon Sep 17 00:00:00 2001 From: Boris Kolpackov Date: Sat, 17 Dec 2016 17:18:16 +0200 Subject: Implement test description language (testscript) --- build2/test/testscript/testscript | 276 ---------------------------- reference/build2/test/testscript/testscript | 276 ++++++++++++++++++++++++++++ 2 files changed, 276 insertions(+), 276 deletions(-) delete mode 100644 build2/test/testscript/testscript create mode 100644 reference/build2/test/testscript/testscript diff --git a/build2/test/testscript/testscript b/build2/test/testscript/testscript deleted file mode 100644 index b61280c..0000000 --- a/build2/test/testscript/testscript +++ /dev/null @@ -1,276 +0,0 @@ -- Test description language (testscript) [feature] - -Requirements -============ - -* Run multiple tests - -* Multiple testscript files (to group related test) - -* Specify exit status (optional, expect 0 by default). - -* Run pre/post test actions (setups/teardowns, e.g., database schema creation). - -* Test input (optional) - -* Test stdout output (to compare to, optional) - -* Test stderr output (to compare to, optional) - -* Test description - -* Create (and automatically clean up) input/output files - -? Variable expansion, looked up on target in buildfile - -* Ability to pass arguments from build files (how for multiple scripts?) - Could use $0, $1, $*. - -* Ability to pass arguments from command line. - -? if/else, loops (we will have ternary operator as part of eval context) - -* Multi-line comments (to disable a chunk of test; #\) - -* Ability to suppress output (/dev/null) - -* Ability not to clean up output (if failed) - -* This should be consistent/uniform with how we handle simple tests without - testscripts (i.e., perhaps they can be treated "as if" they had a one-line - testscript). - -Questions -========= - -* What if the testcript attached at the buildfile level? An executable? What - if the test is not for an executable. I guess we always have the directory - target. - - We could reuse the test variable to specify (list of) scripts: - - exe{hello}: test = test1 test2 - - Or maybe testscript should be a target (usually in src) itself? - -* Do we want a notion of multi-command test (e.g., with pre/post commands)? - Maybe {}? Will need it not to clean up output files too early. Also var - scope. - -* What if we need to define some global variables for the entire script? - -* Ability to include testscripts (for grouping/organization purposes). Also - include common definitions? - -* Ability to redefine $0 (e.g., to src_base to run scripts)? - -* What extension to use for script files? Should it be conventional or - configurable/enforced? - - .t, .test, .ts, .tst, .bt, - - .bts .testscript - - just testscript (but if multiple tests then could be desirable to name them - appropriately). - - Maybe either 'testscript' or '.testscript'? - -Notes -===== - -* If expected exit status is not 0, then probably makes sense to suppress - stderr output if there is no stderr comparison. - -Syntax -====== - -The goal is to be as concise as possible at least for the common cases. - -$0 World # hello World, 0 exit expected -$0 "John Doe" # hello "John Doe" -$0 == 1 # hello, 1 exit expected -$0 != 0 # hello, non-0 exit expected - -$0 World >>EOO -Hello, World! -EOO - -$0 World >"Hello, World!" - -$0 2>>EOE -Usage: $0 -EOE - -$0 --trace <>EOO 2>>EOE # The data should appear in the order specified. -World -EOI -Hello, World! -EOO -13 characters written -EOE - -~ Built-in cat command with auto-cleanup? To create files for tests, etc. - -~ Built-in sed(-like) utility? Note that GCC 4.8 has broken support, - only usable from 4.9. Could be a problem. - -~ We should probably allow assignment of variables and since we will also - lookup those defined in the buildfile, makes sense to use the same semantics - (and perhaps even the parser, like we do for command line). Also value - types. - - So in a sense for each test we will have an "inner" variable scope where - we lookup first. - -~ Could support assigning output of a program to a variable: - - foo = `sed ...` - - The semantics should probably be "as if you had the output literally", - without any of the extra shell splitting nonsense, just the standard - buildfile logic. - - Probably also support piping. - -~ Output file redirects should be relative to out_base. Input? Could actually - track created out files and use that if exists, and src otherwise. What if - the test create a file without a redirect? Will need to "register" it - somehow for auto-cleanup. Maybe &? Also for directories &/. - - Does this mean that any program argument starting with & is recognized to - be a file/directory for auto-cleanup? - -~ Will there be a way to combine stderr and stdout, could be useful sometimes - (e.g., order of output relative to diagnostics). 2>& syntax? - -~ Do we support command piping? - -~ What if we actually do want the output to go to a file, e.g., for a second - command to act on it? Maybe keep shell syntax since familiar? Or reverse it - (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I - like it. What about appending to file - >>>>? Maybe >>>&file? - -~ $0 is target path (works out in case of a directory, but will be out). - -~ #-comment (also for mid-line) - -~ quoting & escaping (need to think and specify). Would be nice to avoid shell - madness. Perhaps only support escaping only inside quoted strings). - -~ If we support variable expansion inside "here doc", what if we need to - specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only - do effective escaping? Otherwise will have to escape backslashes... - -~ Will probably need to support both $foo and $(foo) syntax. Will need to - replicate var name lexing mode form build2. Perhaps use it to avoid - duplication? - -~ line continuation with \ - -~ == != - -~ Specify named variables instead/in addition to arguments - - ...: test = test1 test2 - ...: test.test1.var1 = ... - -~ Perhaps the default variable type should be strings, not names? Thought - we reverse most things pretty well. - -~ CWD should probably be running tests in out_base (so if they create some - files). - -~ Clean up is tricky: sometimes we may want to keep it, but then how to clean - it up later (the test might fail if there is junk present). Maybe create - temp directory in out_base and run there? Will also help with parallel runs. - -~ We will need an ability to execute same command multiple times with some - argument variations. For simple cases the following idiom can be used: - - cmd = a b c - $cmd d e f # Expands to: a b c d e f - v=`$cmd x y z` - - The problem is that only the last arguments (of the last program in pipe) can - be varied. - - What if we can support variable templates with the following syntax: - - # Template variable definition. Same as a regular variable assignement but - # $n variables are not expanded and denotes template parameters. - # - cmd ~= a $1 b | c $2 - - # Instantiations. First n values that follows template variable expansion are - # template arguments. - # - $cmd d e f # Expands to: a d b | c e f - v=`$cmd x y` - - Is this really common, this need to have a piped multi-command and to pass - arguments to more than one of them? Remember, the idea is that tests must - be as direct (literal) as possible. Any kind of indirection (or hiding - things in variables) would just make stuff harder to understand. So maybe - something like this should be just written as: - - a d b | c e f - v=`a x b | c y f` - - But would be nice to see a real use-case that needed this. - -~ How to control script execution from the command line? What about the - followig approach. - - For test operation build2 assumes arguments which follows a special marker - (-) to be intended for a test script target and get assigned to a special - variable ($@). Test script normally just pass them to the program being - tested. In the absense of the marker all unrecognized arguments are assigned - to $@. So effectivelly the marker is required only to disambiguate build2 own - arguments from a test target ones. Test script do not extract any information - from $@ (while can modify it as any other variable). - - To ammend a test script execution the command line variables can be used. - Probably it make sense to require such a variable to be defined in a - buildfile to make sure it is initialized (with a default value). Example: - - buildfile: - - ./: test = test1 - ./: test.test1.c = "" - ./: test.test1.remote = false - ./: test.test1.verbose = false - - test1 script: - - $@ = (verbose ? : "-q") + $@ - cxx_options = (c != "" ? "config.cxx=$c" : ) - - Command line: - - $ b test remote=true c=clang++ verbose=true - --verbose 4 - - To define variable for all target test script we could use a special script - name (*): - - ./: test = test1 test2 - ./: test.*.c = "" - - It is also tempting to extract test script variable values from command line - options, so the above example could be changed in the following way: - - buildfile: - - ./: test = test1 - ./: test.test1.c = "" - ./: test.test1.remote = false - ./: test.test1.verbose = 0 - - test1 script: - - $@ = (verbose == 0 ? "-q" : ) + $@ - cxx_options = (c != "" ? "config.cxx=$c" : ) - - Command line: - - $ b test remote=true c=clang++ - --verbose 4 diff --git a/reference/build2/test/testscript/testscript b/reference/build2/test/testscript/testscript new file mode 100644 index 0000000..1621389 --- /dev/null +++ b/reference/build2/test/testscript/testscript @@ -0,0 +1,276 @@ ++ Test description language (testscript) [feature] + +Requirements +============ + ++ Run multiple tests + ++ Multiple testscript files (to group related test) + ++ Specify exit status (optional, expect 0 by default). + ++ Run pre/post test actions (setups/teardowns, e.g., database schema creation). + ++ Test input (optional) + ++ Test stdout output (to compare to, optional) + ++ Test stderr output (to compare to, optional) + ++ Test description + ++ Create (and automatically clean up) input/output files + ++ Variable expansion, looked up on target in buildfile + ++ Ability to pass arguments from build files (how for multiple scripts?) + Could use $0, $1, $*. + +* Ability to pass arguments from command line. + ++ if/else, loops (we will have ternary operator as part of eval context) + ++ Multi-line comments (to disable a chunk of test; #\) + ++ Ability to suppress output (/dev/null) + ++ Ability not to clean up output (if failed) + +* This should be consistent/uniform with how we handle simple tests without + testscripts (i.e., perhaps they can be treated "as if" they had a one-line + testscript). + +Questions +========= + +* What if the testcript attached at the buildfile level? An executable? What + if the test is not for an executable. I guess we always have the directory + target. + + We could reuse the test variable to specify (list of) scripts: + + exe{hello}: test = test1 test2 + + Or maybe testscript should be a target (usually in src) itself? + +* Do we want a notion of multi-command test (e.g., with pre/post commands)? + Maybe {}? Will need it not to clean up output files too early. Also var + scope. + +* What if we need to define some global variables for the entire script? + +* Ability to include testscripts (for grouping/organization purposes). Also + include common definitions? + +* Ability to redefine $0 (e.g., to src_base to run scripts)? + +* What extension to use for script files? Should it be conventional or + configurable/enforced? + + .t, .test, .ts, .tst, .bt, + + .bts .testscript + + just testscript (but if multiple tests then could be desirable to name them + appropriately). + + Maybe either 'testscript' or '.testscript'? + +Notes +===== + +* If expected exit status is not 0, then probably makes sense to suppress + stderr output if there is no stderr comparison. No, could be unexpected. + +Syntax +====== + +The goal is to be as concise as possible at least for the common cases. + +$0 World # hello World, 0 exit expected +$0 "John Doe" # hello "John Doe" +$0 == 1 # hello, 1 exit expected +$0 != 0 # hello, non-0 exit expected + +$0 World >>EOO +Hello, World! +EOO + +$0 World >"Hello, World!" + +$0 2>>EOE +Usage: $0 +EOE + +$0 --trace <>EOO 2>>EOE # The data should appear in the order specified. +World +EOI +Hello, World! +EOO +13 characters written +EOE + +~ Built-in cat command with auto-cleanup? To create files for tests, etc. + +~ Built-in sed(-like) utility? Note that GCC 4.8 has broken support, + only usable from 4.9. Could be a problem. + +~ We should probably allow assignment of variables and since we will also + lookup those defined in the buildfile, makes sense to use the same semantics + (and perhaps even the parser, like we do for command line). Also value + types. + + So in a sense for each test we will have an "inner" variable scope where + we lookup first. + +~ Could support assigning output of a program to a variable: + + foo = `sed ...` + + The semantics should probably be "as if you had the output literally", + without any of the extra shell splitting nonsense, just the standard + buildfile logic. + + Probably also support piping. + +~ Output file redirects should be relative to out_base. Input? Could actually + track created out files and use that if exists, and src otherwise. What if + the test create a file without a redirect? Will need to "register" it + somehow for auto-cleanup. Maybe &? Also for directories &/. + + Does this mean that any program argument starting with & is recognized to + be a file/directory for auto-cleanup? + +~ Will there be a way to combine stderr and stdout, could be useful sometimes + (e.g., order of output relative to diagnostics). 2>& syntax? + +~ Do we support command piping? + +~ What if we actually do want the output to go to a file, e.g., for a second + command to act on it? Maybe keep shell syntax since familiar? Or reverse it + (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I + like it. What about appending to file - >>>>? Maybe >>>&file? + +~ $0 is target path (works out in case of a directory, but will be out). + +~ #-comment (also for mid-line) + +~ quoting & escaping (need to think and specify). Would be nice to avoid shell + madness. Perhaps only support escaping only inside quoted strings). + +~ If we support variable expansion inside "here doc", what if we need to + specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only + do effective escaping? Otherwise will have to escape backslashes... + +~ Will probably need to support both $foo and $(foo) syntax. Will need to + replicate var name lexing mode form build2. Perhaps use it to avoid + duplication? + +~ line continuation with \ + +~ == != + +~ Specify named variables instead/in addition to arguments + + ...: test = test1 test2 + ...: test.test1.var1 = ... + +~ Perhaps the default variable type should be strings, not names? Thought + we reverse most things pretty well. + +~ CWD should probably be running tests in out_base (so if they create some + files). + +~ Clean up is tricky: sometimes we may want to keep it, but then how to clean + it up later (the test might fail if there is junk present). Maybe create + temp directory in out_base and run there? Will also help with parallel runs. + +~ We will need an ability to execute same command multiple times with some + argument variations. For simple cases the following idiom can be used: + + cmd = a b c + $cmd d e f # Expands to: a b c d e f + v=`$cmd x y z` + + The problem is that only the last arguments (of the last program in pipe) can + be varied. + + What if we can support variable templates with the following syntax: + + # Template variable definition. Same as a regular variable assignement but + # $n variables are not expanded and denotes template parameters. + # + cmd ~= a $1 b | c $2 + + # Instantiations. First n values that follows template variable expansion are + # template arguments. + # + $cmd d e f # Expands to: a d b | c e f + v=`$cmd x y` + + Is this really common, this need to have a piped multi-command and to pass + arguments to more than one of them? Remember, the idea is that tests must + be as direct (literal) as possible. Any kind of indirection (or hiding + things in variables) would just make stuff harder to understand. So maybe + something like this should be just written as: + + a d b | c e f + v=`a x b | c y f` + + But would be nice to see a real use-case that needed this. + +~ How to control script execution from the command line? What about the + followig approach. + + For test operation build2 assumes arguments which follows a special marker + (-) to be intended for a test script target and get assigned to a special + variable ($@). Test script normally just pass them to the program being + tested. In the absense of the marker all unrecognized arguments are assigned + to $@. So effectivelly the marker is required only to disambiguate build2 own + arguments from a test target ones. Test script do not extract any information + from $@ (while can modify it as any other variable). + + To ammend a test script execution the command line variables can be used. + Probably it make sense to require such a variable to be defined in a + buildfile to make sure it is initialized (with a default value). Example: + + buildfile: + + ./: test = test1 + ./: test.test1.c = "" + ./: test.test1.remote = false + ./: test.test1.verbose = false + + test1 script: + + $@ = (verbose ? : "-q") + $@ + cxx_options = (c != "" ? "config.cxx=$c" : ) + + Command line: + + $ b test remote=true c=clang++ verbose=true - --verbose 4 + + To define variable for all target test script we could use a special script + name (*): + + ./: test = test1 test2 + ./: test.*.c = "" + + It is also tempting to extract test script variable values from command line + options, so the above example could be changed in the following way: + + buildfile: + + ./: test = test1 + ./: test.test1.c = "" + ./: test.test1.remote = false + ./: test.test1.verbose = 0 + + test1 script: + + $@ = (verbose == 0 ? "-q" : ) + $@ + cxx_options = (c != "" ? "config.cxx=$c" : ) + + Command line: + + $ b test remote=true c=clang++ - --verbose 4 -- cgit v1.1