summaryrefslogtreecommitdiff
path: root/build2/test
diff options
context:
space:
mode:
authorBoris Kolpackov <boris@codesynthesis.com>2016-12-17 17:18:16 +0200
committerBoris Kolpackov <boris@codesynthesis.com>2016-12-17 17:18:16 +0200
commitdf70b9407714ea9846078c3f727977c501ad2349 (patch)
tree08cf741e2af06c69d38540eb823a3c5286efb16a /build2/test
parentcee59eb9d872acbbf973e8f8f9a05a8c9cb9fb44 (diff)
Implement test description language (testscript)
Diffstat (limited to 'build2/test')
-rw-r--r--build2/test/testscript/testscript276
1 files changed, 0 insertions, 276 deletions
diff --git a/build2/test/testscript/testscript b/build2/test/testscript/testscript
deleted file mode 100644
index b61280c..0000000
--- a/build2/test/testscript/testscript
+++ /dev/null
@@ -1,276 +0,0 @@
-- Test description language (testscript) [feature]
-
-Requirements
-============
-
-* Run multiple tests
-
-* Multiple testscript files (to group related test)
-
-* Specify exit status (optional, expect 0 by default).
-
-* Run pre/post test actions (setups/teardowns, e.g., database schema creation).
-
-* Test input (optional)
-
-* Test stdout output (to compare to, optional)
-
-* Test stderr output (to compare to, optional)
-
-* Test description
-
-* Create (and automatically clean up) input/output files
-
-? Variable expansion, looked up on target in buildfile
-
-* Ability to pass arguments from build files (how for multiple scripts?)
- Could use $0, $1, $*.
-
-* Ability to pass arguments from command line.
-
-? if/else, loops (we will have ternary operator as part of eval context)
-
-* Multi-line comments (to disable a chunk of test; #\)
-
-* Ability to suppress output (/dev/null)
-
-* Ability not to clean up output (if failed)
-
-* This should be consistent/uniform with how we handle simple tests without
- testscripts (i.e., perhaps they can be treated "as if" they had a one-line
- testscript).
-
-Questions
-=========
-
-* What if the testcript attached at the buildfile level? An executable? What
- if the test is not for an executable. I guess we always have the directory
- target.
-
- We could reuse the test variable to specify (list of) scripts:
-
- exe{hello}: test = test1 test2
-
- Or maybe testscript should be a target (usually in src) itself?
-
-* Do we want a notion of multi-command test (e.g., with pre/post commands)?
- Maybe {}? Will need it not to clean up output files too early. Also var
- scope.
-
-* What if we need to define some global variables for the entire script?
-
-* Ability to include testscripts (for grouping/organization purposes). Also
- include common definitions?
-
-* Ability to redefine $0 (e.g., to src_base to run scripts)?
-
-* What extension to use for script files? Should it be conventional or
- configurable/enforced?
-
- .t, .test, .ts, .tst, .bt, <none>
-
- .bts .testscript
-
- just testscript (but if multiple tests then could be desirable to name them
- appropriately).
-
- Maybe either 'testscript' or '<name>.testscript'?
-
-Notes
-=====
-
-* If expected exit status is not 0, then probably makes sense to suppress
- stderr output if there is no stderr comparison.
-
-Syntax
-======
-
-The goal is to be as concise as possible at least for the common cases.
-
-$0 World # hello World, 0 exit expected
-$0 "John Doe" # hello "John Doe"
-$0 == 1 # hello, 1 exit expected
-$0 != 0 # hello, non-0 exit expected
-
-$0 World >>EOO
-Hello, World!
-EOO
-
-$0 World >"Hello, World!"
-
-$0 2>>EOE
-Usage: $0 <name>
-EOE
-
-$0 --trace <<EOI >>EOO 2>>EOE # The data should appear in the order specified.
-World
-EOI
-Hello, World!
-EOO
-13 characters written
-EOE
-
-~ Built-in cat command with auto-cleanup? To create files for tests, etc.
-
-~ Built-in sed(-like) utility? Note that GCC 4.8 has broken <regex> support,
- only usable from 4.9. Could be a problem.
-
-~ We should probably allow assignment of variables and since we will also
- lookup those defined in the buildfile, makes sense to use the same semantics
- (and perhaps even the parser, like we do for command line). Also value
- types.
-
- So in a sense for each test we will have an "inner" variable scope where
- we lookup first.
-
-~ Could support assigning output of a program to a variable:
-
- foo = `sed ...`
-
- The semantics should probably be "as if you had the output literally",
- without any of the extra shell splitting nonsense, just the standard
- buildfile logic.
-
- Probably also support piping.
-
-~ Output file redirects should be relative to out_base. Input? Could actually
- track created out files and use that if exists, and src otherwise. What if
- the test create a file without a redirect? Will need to "register" it
- somehow for auto-cleanup. Maybe &<file>? Also for directories &<dir>/.
-
- Does this mean that any program argument starting with & is recognized to
- be a file/directory for auto-cleanup?
-
-~ Will there be a way to combine stderr and stdout, could be useful sometimes
- (e.g., order of output relative to diagnostics). 2>& syntax?
-
-~ Do we support command piping?
-
-~ What if we actually do want the output to go to a file, e.g., for a second
- command to act on it? Maybe keep shell syntax since familiar? Or reverse it
- (> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I
- like it. What about appending to file - >>>>? Maybe >>>&file?
-
-~ $0 is target path (works out in case of a directory, but will be out).
-
-~ #-comment (also for mid-line)
-
-~ quoting & escaping (need to think and specify). Would be nice to avoid shell
- madness. Perhaps only support escaping only inside quoted strings).
-
-~ If we support variable expansion inside "here doc", what if we need to
- specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only
- do effective escaping? Otherwise will have to escape backslashes...
-
-~ Will probably need to support both $foo and $(foo) syntax. Will need to
- replicate var name lexing mode form build2. Perhaps use it to avoid
- duplication?
-
-~ line continuation with \
-
-~ == !=
-
-~ Specify named variables instead/in addition to arguments
-
- ...: test = test1 test2
- ...: test.test1.var1 = ...
-
-~ Perhaps the default variable type should be strings, not names? Thought
- we reverse most things pretty well.
-
-~ CWD should probably be running tests in out_base (so if they create some
- files).
-
-~ Clean up is tricky: sometimes we may want to keep it, but then how to clean
- it up later (the test might fail if there is junk present). Maybe create
- temp directory in out_base and run there? Will also help with parallel runs.
-
-~ We will need an ability to execute same command multiple times with some
- argument variations. For simple cases the following idiom can be used:
-
- cmd = a b c
- $cmd d e f # Expands to: a b c d e f
- v=`$cmd x y z`
-
- The problem is that only the last arguments (of the last program in pipe) can
- be varied.
-
- What if we can support variable templates with the following syntax:
-
- # Template variable definition. Same as a regular variable assignement but
- # $n variables are not expanded and denotes template parameters.
- #
- cmd ~= a $1 b | c $2
-
- # Instantiations. First n values that follows template variable expansion are
- # template arguments.
- #
- $cmd d e f # Expands to: a d b | c e f
- v=`$cmd x y`
-
- Is this really common, this need to have a piped multi-command and to pass
- arguments to more than one of them? Remember, the idea is that tests must
- be as direct (literal) as possible. Any kind of indirection (or hiding
- things in variables) would just make stuff harder to understand. So maybe
- something like this should be just written as:
-
- a d b | c e f
- v=`a x b | c y f`
-
- But would be nice to see a real use-case that needed this.
-
-~ How to control script execution from the command line? What about the
- followig approach.
-
- For test operation build2 assumes arguments which follows a special marker
- (-) to be intended for a test script target and get assigned to a special
- variable ($@). Test script normally just pass them to the program being
- tested. In the absense of the marker all unrecognized arguments are assigned
- to $@. So effectivelly the marker is required only to disambiguate build2 own
- arguments from a test target ones. Test script do not extract any information
- from $@ (while can modify it as any other variable).
-
- To ammend a test script execution the command line variables can be used.
- Probably it make sense to require such a variable to be defined in a
- buildfile to make sure it is initialized (with a default value). Example:
-
- buildfile:
-
- ./: test = test1
- ./: test.test1.c = ""
- ./: test.test1.remote = false
- ./: test.test1.verbose = false
-
- test1 script:
-
- $@ = (verbose ? : "-q") + $@
- cxx_options = (c != "" ? "config.cxx=$c" : )
-
- Command line:
-
- $ b test remote=true c=clang++ verbose=true - --verbose 4
-
- To define variable for all target test script we could use a special script
- name (*):
-
- ./: test = test1 test2
- ./: test.*.c = ""
-
- It is also tempting to extract test script variable values from command line
- options, so the above example could be changed in the following way:
-
- buildfile:
-
- ./: test = test1
- ./: test.test1.c = ""
- ./: test.test1.remote = false
- ./: test.test1.verbose = 0
-
- test1 script:
-
- $@ = (verbose == 0 ? "-q" : ) + $@
- cxx_options = (c != "" ? "config.cxx=$c" : )
-
- Command line:
-
- $ b test remote=true c=clang++ - --verbose 4