diff options
Diffstat (limited to 'doc/asciidoc/doc/testasciidoc.txt')
| -rw-r--r-- | doc/asciidoc/doc/testasciidoc.txt | 231 |
1 files changed, 231 insertions, 0 deletions
diff --git a/doc/asciidoc/doc/testasciidoc.txt b/doc/asciidoc/doc/testasciidoc.txt new file mode 100644 index 0000000..1d9a1ea --- /dev/null +++ b/doc/asciidoc/doc/testasciidoc.txt @@ -0,0 +1,231 @@ +AsciiDoc Tests +============== + +AsciiDoc includes 'testasciidoc', a Python script runs a set of +AsciiDoc conformance tests. 'testasciidoc' runs through a list of +AsciiDoc source files, generates backend outputs and then compares +them to expected result files. The whole process is driven by a +configuration file containing a list of user configurable test +specifications. + + +Rationale +--------- +When modifying AsciiDoc configuration files or source code it's very +easy to introduce regression errors. 'testasciidoc' is a tool for +writing many and varied test cases that can be run after changes have +been made in order to verify that existing behavior has not been +broken. + +The 'testasciidoc' 'update' command automates the (re)generation of +expected test result data. Result data regeneration has to be +performed after changes to test case source files or after legitimate +changes to the AsciiDoc output formats -- doing this manually is +prohibitively tedious. + + +Running testasciidoc +-------------------- +The `testasciidoc.py` script and the default `testasciidoc.conf` +configuration file are located in the AsciiDoc distribution `tests` +directory. + +To view the command usage run: + +--------------------------------------------------------------------- +$ python tests/testasciidoc.py +Usage: testasciidoc.py [OPTIONS] COMMAND + +Run AsciiDoc conformance tests specified in configuration FILE. + +Commands: + list List tests + run [NUMBER] [BACKEND] Execute tests + update [NUMBER] [BACKEND] Regenerate and update test data + +Options: + -f, --conf-file=CONF_FILE + Use configuration file CONF_FILE (default configuration file is + testasciidoc.conf in testasciidoc.py directory) + --force + Update all test data overwriting existing data +--------------------------------------------------------------------- + +To view the list of tests in the default `testasciidoc.conf` +configuration file run the 'list' command: + +--------------------------------------------------------------------- +$ python tests/testasciidoc.py list +1: Test cases +2: Tables +3: Source highlighter +4: Example article +5: Example book +6: Example multi-part book +7: !User Guide +--------------------------------------------------------------------- + +The '!' prefix signals that a test is currently disabled. + +Before running the tests you will need to regenerate the expected +outputs by running the 'update' command: + +--------------------------------------------------------------------- +$ python tests/testasciidoc.py update +WRITING: tests/data/testcases-html4.html +WRITING: tests/data/testcases-xhtml11.html +WRITING: tests/data/testcases-docbook.xml + : +WRITING: tests/data/book-multi-docbook.xml +--------------------------------------------------------------------- + +Now you can run the tests: + +--------------------------------------------------------------------- +$ python tests/testasciidoc.py run +1: Test cases +SOURCE: asciidoc: tests/data/testcases.txt +PASSED: html4: tests/data/testcases-html4.html +PASSED: xhtml11: tests/data/testcases-xhtml11.html +PASSED: docbook: tests/data/testcases-docbook.xml + : +6: Example multi-part book +SOURCE: asciidoc: doc/book-multi.txt +PASSED: html4: tests/data/book-multi-html4.html +PASSED: xhtml11: tests/data/book-multi-xhtml11.html +PASSED: docbook: tests/data/book-multi-docbook.xml + +TOTAL PASSED: 18 +--------------------------------------------------------------------- + +[NOTE] +===================================================================== +- 'testasciidoc' requires AsciiDoc is located in your environment + search path, if not you'll need to set the `ASCIIDOC_PY` environment + variable to point to `asciidoc.py`. +- If you don't have GNU source-highlight installed you should disable + the 'Tables' and 'Source highlighter' tests in the + `tests/testasciidoc.conf` configuration file. +===================================================================== + + +testasciidoc commands +--------------------- +'list':: + List test numbers and titles. A '!' title prefix signals that a + test is currently disabled. + +'run':: + Read and execute tests from the test configuration file. A test + specifies AsciiDoc test case source file and command options. The + test compares generated outputs to expected outputs and any + differences displayed as a diff. You can run selected tests by + specifying the test number and/or backend after the `run` command. + Examples: + + python tests/testasciidoc.py run + python tests/testasciidoc.py run 3 + python tests/testasciidoc.py run html4 + python tests/testasciidoc.py run 3 html4 + +'update':: + Generates and updates missing and out of date test output data + files, this eliminates one of the most time consuming aspect of test + management. Use the `--force` option to force updates. + Examples: + + python tests/testasciidoc.py update + python tests/testasciidoc.py --force update 4 + +NOTE: You can run or update disabled tests by explicitly specifying +the test number. + + +Test configuration file +----------------------- +The tests configuration file format consists of one or more test specs +separated by a line of one or more percent characters. Each test spec +consists of an optional test title and description followed by one or +more optional directives (a directive starts with a percent +character). A 'directive' consists begins with a line containing a '%' +character followed by a directive name followed by zero or more lines +of directive data. + +Test spec format +~~~~~~~~~~~~~~~~ +--------------------------------------------------------------------- +Optional test title +Optional test description... + +% name +Optional base output file name. Defaults to base source file name. + +% source +AsciiDoc source file name. + +% backends +Optional list of backends to be tested(default is all backends). +% options +Optional list of command-line option tuples. + +% attributes +Optional dictionary of attribute values. + +--------------------------------------------------------------------- + +Example test spec: + +--------------------------------------------------------------------- +Example book + +% options +['--section-numbers',('--doctype','book')] + +% attributes +# Exclude date from document footer. +{'docdate':None} + +% source +../doc/book.txt +--------------------------------------------------------------------- + +TIP: Take a look at the default `tests/testasciidoc.conf` +configuration file that comes with AsciiDoc. + +- Tests can be disabled by prefixing the test title with an + exclamation '!' character. +- All relative file names are relative to the configuration file + directory. +- Multiple tests must by separated by a `%` separator line (one or + more percent characters). +- Lines starting with a percent character specify a test 'directive' + and may be followed by zero or more lines of directive data. +- Directive data lines cannot start with a percent character. +- Lines starting with a `#` hash character are ignored. +- The 'source' directive data is a single line containing the + AsciiDoc source file name. +- The 'options' directive data is a Python list of `(name,value)` + tuples specifying AsciiDoc command-line options. A string item is + equivalent to a `(name,None)` tuple. +- The 'attributes' directive data specifies a Python dictionary + containing AsciiDoc attributes to be passed to AsciiDoc. + +globals directive +~~~~~~~~~~~~~~~~~ +An optional 'globals' directive can precede all test specs, the +globals directive data is a Python dictionary containing global +values. Currently the only global is 'datadir', the directory +containing expected output files (defaults to configuration file +directory). For example: + +--------------------------------------------------------------------- +% globals +{'datadir': 'data'} +--------------------------------------------------------------------- + +Expected output test data files are stored in the 'datadir' and are +named after the corresponding AsciiDoc input source file. For example +if the AsciiDoc file name is `article.txt` then the corresponding +backend output files will be `article-html4.html`, +`article-xhtml11.html`, `article-docbook.xml` (stored in the 'datadir' +directory). |